[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 820122246 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002378] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009135] LSM: Security Framework initializing [ 0.010117] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.012054] *** VALIDATE selinux *** [ 0.021593] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026291] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029076] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.032288] *** VALIDATE proc *** [ 0.034086] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036262] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044675] debug: unmapping init [mem 0xffffffffa8259000-0xffffffffa8260fff] [ 0.047000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047830] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048028] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050020] ... generic registers: 4 [ 0.051019] ... value mask: 0000ffffffffffff [ 0.052018] ... max period: 00007fffffffffff [ 0.053021] ... fixed-purpose events: 3 [ 0.054018] ... event mask: 000000070000000f [ 0.055451] rcu: Hierarchical SRCU implementation. [ 0.057612] smp: Bringing up secondary CPUs ... [ 0.058603] x86: Booting SMP configuration: [ 0.059026] .... node #0, CPUs: #1 #2 #3 [ 0.067376] smp: Brought up 1 node, 4 CPUs [ 0.069012] smpboot: Max logical packages: 1 [ 0.070017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.108300] node 0 deferred pages initialised in 36ms [ 0.111665] devtmpfs: initialized [ 0.112225] x86/mm: Memory block size: 128MB [ 0.115372] gcov: version magic: 0x41383552 [ 0.117376] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.118082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.119262] pinctrl core: initialized pinctrl subsystem [ 0.120426] [ 0.121008] ************************************************************* [ 0.122013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.123013] ** ** [ 0.124013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.125015] ** ** [ 0.126012] ** This means that this kernel is built to expose internal ** [ 0.127016] ** IOMMU data structures, which may compromise security on ** [ 0.128012] ** your system. ** [ 0.129010] ** ** [ 0.130016] ** If you see this message and you are not debugging the ** [ 0.131012] ** kernel, report this immediately to your vendor! ** [ 0.132014] ** ** [ 0.133014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.134013] ************************************************************* [ 0.135704] NET: Registered protocol family 16 [ 0.136673] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.137067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.138062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.139662] cpuidle: using governor menu [ 0.142061] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.145468] PCI: Using configuration type 1 for base access [ 0.147275] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.157376] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.158016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.162126] cryptd: max_cpu_qlen set to 1000 [ 0.163785] ACPI: Added _OSI(Module Device) [ 0.164010] ACPI: Added _OSI(Processor Device) [ 0.165000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.168015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.173880] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.181661] ACPI: Interpreter enabled [ 0.184062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.186011] ACPI: Using IOAPIC for interrupt routing [ 0.188127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.192434] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.205636] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.207043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.211027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.215075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.220603] acpiphp: Slot [2] registered [ 0.221116] acpiphp: Slot [5] registered [ 0.223115] acpiphp: Slot [6] registered [ 0.224101] acpiphp: Slot [7] registered [ 0.226122] acpiphp: Slot [8] registered [ 0.228134] acpiphp: Slot [9] registered [ 0.230107] acpiphp: Slot [10] registered [ 0.232139] acpiphp: Slot [3] registered [ 0.233164] acpiphp: Slot [4] registered [ 0.235106] acpiphp: Slot [11] registered [ 0.237104] acpiphp: Slot [12] registered [ 0.239107] acpiphp: Slot [13] registered [ 0.240103] acpiphp: Slot [14] registered [ 0.242104] acpiphp: Slot [15] registered [ 0.244128] acpiphp: Slot [16] registered [ 0.246157] acpiphp: Slot [17] registered [ 0.247234] acpiphp: Slot [18] registered [ 0.249093] acpiphp: Slot [19] registered [ 0.251085] acpiphp: Slot [20] registered [ 0.252298] acpiphp: Slot [21] registered [ 0.254121] acpiphp: Slot [22] registered [ 0.256100] acpiphp: Slot [23] registered [ 0.257226] acpiphp: Slot [24] registered [ 0.259077] acpiphp: Slot [25] registered [ 0.261101] acpiphp: Slot [26] registered [ 0.263121] acpiphp: Slot [27] registered [ 0.264225] acpiphp: Slot [28] registered [ 0.266149] acpiphp: Slot [29] registered [ 0.268148] acpiphp: Slot [30] registered [ 0.269106] acpiphp: Slot [31] registered [ 0.271052] PCI host bridge to bus 0000:00 [ 0.273018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.276027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.279017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.282018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.285111] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.288021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.290147] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.293040] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.295000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.305013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.310050] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.312016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.315016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.318014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.320590] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.323698] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.326036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.328814] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.333012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.348014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.353012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.358918] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.384023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.424025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.436169] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.444022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.451023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.468026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.495862] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.507021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.513019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.528031] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.540978] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.548020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.562023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.581027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.597607] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.614023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.623023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.627551] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.630036] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.634026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.652034] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.666644] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.669543] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.672425] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.675352] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.678266] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.685035] iommu: Default domain type: Passthrough [ 0.686630] SCSI subsystem initialized [ 0.688207] ACPI: bus type USB registered [ 0.690190] usbcore: registered new interface driver usbfs [ 0.693175] usbcore: registered new interface driver hub [ 0.694107] usbcore: registered new device driver usb [ 0.697234] pps_core: LinuxPPS API ver. 1 registered [ 0.699013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.702066] PTP clock support registered [ 0.705075] EDAC MC: Ver: 3.0.0 [ 0.707136] PCI: Using ACPI for IRQ routing [ 0.708512] NetLabel: Initializing [ 0.710020] NetLabel: domain hash size = 128 [ 0.711008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.712077] NetLabel: unlabeled traffic allowed by default [ 0.714146] vgaarb: loaded [ 0.716330] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.717023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.726453] clocksource: Switched to clocksource kvm-clock [ 0.845905] VFS: Disk quotas dquot_6.6.0 [ 0.847253] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.853901] *** VALIDATE ramfs *** [ 0.857848] *** VALIDATE hugetlbfs *** [ 0.859216] pnp: PnP ACPI init [ 0.861422] pnp: PnP ACPI: found 6 devices [ 0.877420] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.880133] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.881969] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.883795] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.885812] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.887902] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.890357] NET: Registered protocol family 2 [ 0.892588] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.897148] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.900554] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.906166] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.909312] TCP: Hash tables configured (established 65536 bind 65536) [ 0.912122] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.914970] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.917688] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.920669] NET: Registered protocol family 1 [ 0.923282] RPC: Registered named UNIX socket transport module. [ 0.925066] RPC: Registered udp transport module. [ 0.926698] RPC: Registered tcp transport module. [ 0.928063] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.930513] NET: Registered protocol family 44 [ 0.932349] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.934079] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.935910] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.937958] PCI: CLS 0 bytes, default 64 [ 0.939424] Unpacking initramfs... [ 2.468345] debug: unmapping init [mem 0xffff8ed9bcc54000-0xffff8ed9bffbffff] [ 2.472398] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.474494] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.477147] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.092844] Initialise system trusted keyrings [ 3.094741] Key type blacklist registered [ 3.096700] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.106350] zbud: loaded [ 3.109469] *** VALIDATE nfs *** [ 3.110822] *** VALIDATE nfs4 *** [ 3.113374] pstore: using deflate compression [ 3.118588] Platform Keyring initialized [ 3.282463] NET: Registered protocol family 38 [ 3.284639] Key type asymmetric registered [ 3.286390] Asymmetric key parser 'x509' registered [ 3.288559] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.292050] io scheduler mq-deadline registered [ 3.294048] io scheduler kyber registered [ 3.295876] io scheduler bfq registered [ 3.297946] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.301750] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.304929] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.308135] ACPI: Power Button [PWRF] [ 3.313826] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.321714] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.366825] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.664241] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.766692] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.796202] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.828409] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.884542] Non-volatile memory driver v1.3 [ 3.885921] Linux agpgart interface v0.103 [ 3.930836] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.935752] vda: detected capacity change from 0 to 76779520 [ 4.070373] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.073898] vdb: detected capacity change from 0 to 1073741824 [ 4.103160] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.156092] vdc: detected capacity change from 0 to 2621440000 [ 4.344611] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.347373] vdd: detected capacity change from 0 to 2621440000 [ 4.407760] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.411277] vde: detected capacity change from 0 to 4294967296 [ 4.431670] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.435729] vdf: detected capacity change from 0 to 4294967296 [ 4.443578] libphy: Fixed MDIO Bus: probed [ 4.475707] usbcore: registered new interface driver usbserial_generic [ 4.478264] usbserial: USB Serial support registered for generic [ 4.482124] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.502768] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.505271] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.508517] mousedev: PS/2 mouse device common for all mice [ 4.512543] rtc_cmos 00:05: RTC can wake from S4 [ 4.517291] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.522397] rtc_cmos 00:05: registered as rtc0 [ 4.527023] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.530123] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.533890] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.538037] intel_pstate: CPU model not supported [ 4.541480] hid: raw HID events driver (C) Jiri Kosina [ 4.544025] usbcore: registered new interface driver usbhid [ 4.546477] usbhid: USB HID core driver [ 4.548364] drop_monitor: Initializing network drop monitor service [ 4.551055] Initializing XFRM netlink socket [ 4.553302] NET: Registered protocol family 10 [ 4.556432] Segment Routing with IPv6 [ 4.557602] NET: Registered protocol family 17 [ 4.560063] mpls_gso: MPLS GSO support [ 4.564741] RAS: Correctable Errors collector initialized. [ 4.567072] AVX version of gcm_enc/dec engaged. [ 4.568560] AES CTR mode by8 optimization enabled [ 4.660579] sched_clock: Marking stable (4660495229, 0)->(5837837487, -1177342258) [ 4.665767] registered taskstats version 1 [ 4.668247] Loading compiled-in X.509 certificates [ 4.670121] zswap: loaded using pool lzo/zbud [ 4.698329] Key type big_key registered [ 4.714776] Key type encrypted registered [ 4.717690] ima: No TPM chip found, activating TPM-bypass! [ 4.720356] ima: Allocated hash algorithm: sha1 [ 4.723259] ima: No architecture policies found [ 4.725429] evm: Initialising EVM extended attributes: [ 4.728639] evm: security.selinux [ 4.729867] evm: security.ima [ 4.731444] evm: security.capability [ 4.734212] evm: HMAC attrs: 0x1 [ 4.738258] rtc_cmos 00:05: setting system clock to 2026-09-08 06:57:55 UTC (1788850675) [ 4.748892] debug: unmapping init [mem 0xffffffffa9203000-0xffffffffa93fffff] [ 4.753855] debug: unmapping init [mem 0xffffffffa7f82000-0xffffffffa8258fff] [ 4.765604] Write protecting the kernel read-only data: 28672k [ 4.775250] debug: unmapping init [mem 0xffffffffa6603000-0xffffffffa67fffff] [ 4.781463] debug: unmapping init [mem 0xffffffffa6f14000-0xffffffffa6ffffff] [ 4.835541] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.847413] systemd[1]: Detected virtualization kvm. [ 4.849531] systemd[1]: Detected architecture x86-64. [ 4.851834] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.880620] systemd[1]: No hostname configured. [ 4.882976] systemd[1]: Set hostname to . [ 4.885802] random: systemd: uninitialized urandom read (16 bytes read) [ 4.889356] systemd[1]: Initializing machine ID from random generator. [ 5.007276] random: ln: uninitialized urandom read (6 bytes read) [ 5.219895] random: systemd: uninitialized urandom read (16 bytes read) [ 5.223239] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.238867] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.246736] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.937770] device-mapper: uevent: version 1.0.3 [ 6.940455] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 8.636876] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.033667] virtio_net virtio0 ens2: renamed from eth0 [ 9.491044] scsi host0: ata_piix [ 9.571840] scsi host1: ata_piix [ 9.617569] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.628500] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.269485] random: crng init done [ 15.271957] random: 7 urandom warning(s) missed due to ratelimiting [ 17.437386] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.026512] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.519929] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.371065] SELinux: Disabled at runtime. [ 24.490603] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.507934] systemd[1]: Detected virtualization kvm. [ 24.512436] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.907712] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.929592] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.955813] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.980690] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.009190] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.057868] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.089603] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ 28.250848] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 29.534875] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.329105] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.446087] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.263457] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.482263] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 38.408272] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 39.677973] NFS: Registering the id_resolver key type [ 39.686123] Key type id_resolver registered [ 39.689096] Key type id_legacy registered [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg241-server login: [ 99.833756] libcfs: loading out-of-tree module taints kernel. [ 99.869522] Key type ._llcrypt registered [ 99.871413] Key type .llcrypt registered [ 99.962880] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 117.398509] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 118.679880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 118.690496] alg: No test for adler32 (adler32-zlib) [ 120.067188] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 120.715375] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 122.431439] Key type lgssc registered [ 124.195440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 141.783720] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 182.211152] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 195.432486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 195.494437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.099057] hrtimer: interrupt took 2176488 ns [ 196.785335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.831640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.948606] Lustre: lustre-MDT0000: new disk, initializing [ 197.083060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.108781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.855549] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.492204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.671534] Lustre: 6534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.733941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.742676] Lustre: Skipped 1 previous similar message [ 217.046590] Lustre: lustre-MDT0001: new disk, initializing [ 217.160940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.230043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.253974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 223.162558] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.112975] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 242.147231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.487466] Lustre: lustre-OST0000: new disk, initializing [ 242.491140] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 242.497720] Lustre: 8471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.565861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 249.453185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 249.474126] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 249.601254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 250.762717] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.859184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.084817] Lustre: lustre-OST0001: new disk, initializing [ 267.091194] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 267.097976] Lustre: 9545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 267.190615] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.924590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 272.945018] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 273.063455] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 274.881567] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.528728] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 300.134328] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 306.838851] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing check_logdir /tmp/testlogs/ [ 312.558525] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing yml_node [ 317.106430] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 319.478634] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 322.275430] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 323.868147] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 03:03:13 EDT 2026 [ 340.663306] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 342.184503] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 359.908673] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.910585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.917748] Lustre: Skipped 1 previous similar message [ 365.804711] Lustre: server umount lustre-MDT0000 complete [ 369.606996] LustreError: 6524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851040 with bad export cookie 13304553417107310467 [ 369.613277] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 369.623168] LustreError: 6524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 370.155432] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.164131] LustreError: 6540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.168301] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 370.168311] Lustre: Skipped 3 previous similar messages [ 370.175871] Lustre: Skipped 3 previous similar messages [ 370.195195] LustreError: 6540:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 374.184550] LustreError: 6540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.187562] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.210861] Lustre: Skipped 2 previous similar messages [ 376.068671] Lustre: server umount lustre-MDT0001 complete [ 379.750647] Lustre: server umount lustre-OST0000 complete [ 384.116323] Lustre: server umount lustre-OST0001 complete [ 393.911801] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 405.265348] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 446.596988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.989762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 447.050724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 447.214452] Lustre: lustre-MDT0000: new disk, initializing [ 447.394532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.428296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 453.725626] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 465.326462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.537966] Lustre: 15098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 465.686631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 465.695469] Lustre: Skipped 1 previous similar message [ 465.810890] Lustre: lustre-MDT0001: new disk, initializing [ 465.989168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.062091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 466.078787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 471.167368] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 482.887817] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 488.459692] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 496.776192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 497.162896] Lustre: lustre-OST0000: new disk, initializing [ 497.165954] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 497.173132] Lustre: 17069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 497.270832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.816382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 498.820255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 499.066480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 505.404619] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 518.862955] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 525.446592] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 525.582271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 531.070306] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 531.392531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 535.008576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 535.017041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.021936] LustreError: Skipped 1 previous similar message [ 535.066674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 535.080489] Lustre: Skipped 1 previous similar message [ 535.093689] Lustre: Skipped 1 previous similar message [ 539.674845] Lustre: server umount lustre-OST0000 complete [ 545.249968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.254247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 545.257729] Lustre: Skipped 1 previous similar message [ 545.263697] Lustre: Skipped 1 previous similar message [ 550.194419] Lustre: server umount lustre-MDT0000 complete [ 550.388325] LustreError: 15112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.413841] LustreError: 15112:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 553.867586] LustreError: 15093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851224 with bad export cookie 13304553417107312000 [ 553.870751] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 553.884027] LustreError: 15093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 554.267211] Lustre: server umount lustre-MDT0001 complete [ 561.053373] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 03:07:10 (1788851230) [ 579.726241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.446225] LustreError: 19943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 580.565590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.574696] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 585.719445] LustreError: 19944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.401052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.178444] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 609.363942] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 615.932502] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 618.798535] Lustre: Failing over lustre-MDT0000 [ 619.209336] Lustre: server umount lustre-MDT0000 complete [ 619.883806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.897712] LustreError: 20671:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.904227] Lustre: Skipped 1 previous similar message [ 619.937413] LustreError: 20671:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 635.360499] Lustre: 3652:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788851290/real 1788851290] req@ffff8eda10c25500 x1875746007190784/t0(0) o400->MGC192.168.202.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788851306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 635.400053] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.522195] LustreError: 19944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.539710] LustreError: 19944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 639.762561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.629272] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb8a344d7a1e41306 [ 646.643346] Lustre: MGC192.168.202.141@tcp: Connection restored to 0@lo (at 0@lo) [ 646.991527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 647.004300] Lustre: Skipped 1 previous similar message [ 647.073655] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 652.257483] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 652.278740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 652.311441] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 652.821327] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 663.498967] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 667.620346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 667.634641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.654387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 673.830225] Lustre: server umount lustre-MDT0000 complete [ 674.792811] LustreError: 19944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 674.807310] LustreError: 19944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 677.283284] LustreError: 19923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851348 with bad export cookie 13304553417107313414 [ 677.297351] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.653756] Lustre: server umount lustre-MDT0001 complete [ 685.340990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.746906] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 698.077470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.349731] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 709.764705] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 716.245536] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 718.625245] Lustre: Failing over lustre-MDT0000 [ 718.828785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 718.841197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.870465] Lustre: Skipped 1 previous similar message [ 719.041804] Lustre: server umount lustre-MDT0000 complete [ 737.248986] Lustre: 3652:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788851391/real 1788851391] req@ffff8eda3f077b80 x1875746007220864/t0(0) o400->MGC192.168.202.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788851407 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 737.264280] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 738.290246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.371067] LustreError: 23278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 742.405686] LustreError: 23278:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 747.794537] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 752.022582] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 753.131912] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 753.144850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 753.164468] Lustre: Skipped 1 previous similar message [ 753.191655] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 762.904805] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 768.132352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 768.149418] Lustre: Skipped 1 previous similar message [ 768.159657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 768.165136] Lustre: Skipped 3 previous similar messages [ 774.757897] Lustre: server umount lustre-MDT0000 complete [ 780.123037] LustreError: 23264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851450 with bad export cookie 13304553417107314352 [ 780.126293] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 780.149909] LustreError: 23264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 780.937601] Lustre: server umount lustre-MDT0001 complete [ 791.672072] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 03:11:01 (1788851461) [ 807.805295] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 819.561385] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 865.257340] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 877.320044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.599965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 877.629353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 877.686797] Lustre: lustre-MDT0000: new disk, initializing [ 877.743182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.750264] Lustre: Skipped 3 previous similar messages [ 877.762777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 883.633817] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 895.898793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.081378] Lustre: 29662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 896.629670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 902.278979] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 911.504546] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 917.174134] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 924.393994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 924.642898] Lustre: 31630:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 925.886405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 925.926788] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 926.008216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 926.022762] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 931.132905] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 942.564496] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 949.683041] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 977.389223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 977.391547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.395661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 977.395671] Lustre: Skipped 2 previous similar messages [ 977.411223] LustreError: Skipped 1 previous similar message [ 977.444376] Lustre: Skipped 2 previous similar messages [ 981.976528] Lustre: server umount lustre-OST0000 complete [ 993.279720] Lustre: server umount lustre-MDT0000 complete [ 997.859702] LustreError: 29673:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.884812] LustreError: 29673:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 998.652179] LustreError: 29654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851669 with bad export cookie 13304553417107314758 [ 998.655555] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 998.669290] LustreError: 29654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 999.385360] Lustre: server umount lustre-MDT0001 complete [ 1014.355282] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1018.425362] Key type lgssc unregistered [ 1018.906854] LNet: 33611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.922543] LNetError: 33611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.950293] LNet: Removed LNI 192.168.202.141@tcp [ 1020.139266] Key type .llcrypt unregistered [ 1020.143388] Key type ._llcrypt unregistered [ 1047.538792] Key type ._llcrypt registered [ 1047.541701] Key type .llcrypt registered [ 1047.656811] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1048.720480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.913725] alg: No test for adler32 (adler32-zlib) [ 1050.161756] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1050.518040] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1052.239273] Key type lgssc registered [ 1053.344645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1061.450374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1061.487232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.032813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.702979] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1072.675789] LustreError: 35024:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.690869] LustreError: 35024:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1077.525728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.729467] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1077.934170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1083.442389] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1091.764911] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1097.339166] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1104.114491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1104.554352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.656879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1110.925465] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1118.423182] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1122.993934] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1136.695799] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 03:16:46 (1788851806) [ 1138.560509] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1140.258741] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1146.943611] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 03:16:57 (1788851817) [ 1155.504989] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 03:17:05 (1788851825) [ 1156.535290] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1165.563630] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1165.568227] Lustre: Skipped 2 previous similar messages [ 1167.959284] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 03:17:17 (1788851837) [ 1170.398946] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1170.413314] Lustre: Skipped 2 previous similar messages [ 1178.884666] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1178.893382] Lustre: Skipped 1 previous similar message [ 1195.556409] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 1195.569674] Lustre: Skipped 8 previous similar messages [ 1198.101378] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 03:17:47 (1788851867) [ 1201.553369] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1206.122828] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1216.686783] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 03:18:06 (1788851886) [ 1230.842678] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1230.848198] Lustre: Skipped 4 previous similar messages [ 1232.687870] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 03:18:22 (1788851902) [ 1251.296668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1251.309892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.337195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1251.345380] Lustre: Skipped 1 previous similar message [ 1256.417607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1256.434168] Lustre: Skipped 1 previous similar message [ 1257.294482] Lustre: server umount lustre-OST0000 complete [ 1261.537711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.549511] Lustre: Skipped 1 previous similar message [ 1261.555053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.657562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.669077] Lustre: Skipped 2 previous similar messages [ 1275.359194] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1275.594979] Lustre: server umount lustre-MDT0000 complete [ 1276.906102] LustreError: 35024:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.948966] LustreError: 35024:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1280.905309] LustreError: 38937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851951 with bad export cookie 1202281287767542164 [ 1280.908450] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1280.912397] LustreError: 38937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1281.575041] Lustre: server umount lustre-MDT0001 complete [ 1295.218717] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1298.840317] Key type lgssc unregistered [ 1299.225349] LNet: 42880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.236482] LNetError: 42880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1299.262036] LNet: Removed LNI 192.168.202.141@tcp [ 1300.386559] Key type .llcrypt unregistered [ 1300.389678] Key type ._llcrypt unregistered [ 1321.412320] Key type ._llcrypt registered [ 1321.414183] Key type .llcrypt registered [ 1321.530339] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 1337.857808] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1338.921337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1339.106104] alg: No test for adler32 (adler32-zlib) [ 1340.172747] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1340.352450] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1342.031279] Key type lgssc registered [ 1343.444149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1385.813367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1385.850451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.039481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1387.066719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1387.119211] Lustre: lustre-MDT0000: new disk, initializing [ 1387.187270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.203354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1392.398112] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1403.387529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1403.604659] Lustre: 46035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1403.711722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1403.716812] Lustre: Skipped 1 previous similar message [ 1403.776790] Lustre: lustre-MDT0001: new disk, initializing [ 1403.866812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1403.906772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1403.918496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1409.644579] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1423.954794] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1434.828563] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1445.244492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1445.994618] Lustre: lustre-OST0000: new disk, initializing [ 1446.016665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1446.052906] Lustre: 48003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1446.285755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.116912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1453.128988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1453.197164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1453.585876] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1464.644757] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1468.973615] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1469.095757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1475.851890] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1476.173596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1479.143502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1479.144851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.157776] LustreError: Skipped 1 previous similar message [ 1479.165784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1479.172200] Lustre: Skipped 1 previous similar message [ 1483.755606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1483.767101] Lustre: Skipped 1 previous similar message [ 1485.306543] Lustre: server umount lustre-OST0000 complete [ 1490.928769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1490.960702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.994358] Lustre: Skipped 1 previous similar message [ 1491.010546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1493.988024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1493.990671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.182735] Lustre: server umount lustre-MDT0000 complete [ 1501.637312] LustreError: 46027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852172 with bad export cookie 13490760743300808353 [ 1501.660704] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1501.661592] LustreError: 46027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1502.136786] Lustre: server umount lustre-MDT0001 complete [ 1513.413723] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 03:23:02 (1788852182) [ 1520.037413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.424891] LustreError: 50465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1520.444431] LustreError: 50465:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1520.505624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1525.747045] LustreError: 50466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.083823] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1530.851796] LustreError: 50465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.838491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.971304] LustreError: 50466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1536.235168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1542.135403] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1549.895794] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1556.091724] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1563.544470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1563.956579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1570.591373] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1579.483932] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1584.199246] LustreError: 52994:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1585.167848] Lustre: Modifying parameter general.timeout=1 in log params [ 1586.792627] Lustre: Modifying parameter general.timeout=3 in log params [ 1586.798647] Lustre: Skipped 1 previous similar message [ 1588.838976] Lustre: Modifying parameter general.timeout=5 in log params [ 1588.842837] Lustre: Skipped 1 previous similar message [ 1594.016410] Lustre: Modifying parameter general.timeout=11 in log params [ 1594.021840] Lustre: Skipped 5 previous similar messages [ 1602.470783] Lustre: Modifying parameter general.timeout=20 in log params [ 1602.475782] Lustre: Skipped 8 previous similar messages [ 1619.258413] Lustre: Modifying parameter general.timeout=36 in log params [ 1619.266319] Lustre: Skipped 15 previous similar messages [ 1651.445780] Lustre: Modifying parameter general.timeout=68 in log params [ 1651.453959] Lustre: Skipped 31 previous similar messages [ 1720.920523] Lustre: Disabling parameter general.timeout= in log params [ 1720.927981] Lustre: Skipped 61 previous similar messages [ 1724.642197] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 03:26:34 (1788852394) [ 1727.969035] Lustre: 59947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1730.266122] Lustre: 60047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1734.235274] Lustre: 60256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1734.266619] Lustre: 60256:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1738.469913] Lustre: 60468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1738.476912] Lustre: 60468:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1766.243971] Lustre: 61837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1766.266628] LustreError: 61837:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1770.127992] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 03:27:20 (1788852440) [ 1782.763176] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.770763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1782.800356] Lustre: Skipped 1 previous similar message [ 1782.812201] Lustre: Skipped 2 previous similar messages [ 1787.363665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1787.377486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1789.001189] Lustre: server umount lustre-MDT0000 complete [ 1792.442065] LustreError: 54282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852463 with bad export cookie 13490760743300809235 [ 1792.442629] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1792.461959] LustreError: 54282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1792.826784] Lustre: server umount lustre-MDT0001 complete [ 1809.230548] Lustre: server umount lustre-OST0000 complete [ 1842.189541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.480665] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1842.721255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.253145] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1859.805836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.964052] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1860.031916] Lustre: 64367:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1867.079065] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1877.099831] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1883.825728] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1890.202424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1890.405838] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1890.718449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.721339] Lustre: Skipped 1 previous similar message [ 1897.109314] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 1905.709949] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1907.766960] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1907.773969] Lustre: Skipped 23 previous similar messages [ 1907.819436] Lustre: 66937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1916.898087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1916.912267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.952413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1916.967315] Lustre: Skipped 3 previous similar messages [ 1922.047527] Lustre: server umount lustre-OST0000 complete [ 1927.136369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1927.143047] LustreError: Skipped 1 previous similar message [ 1933.207727] Lustre: server umount lustre-MDT0000 complete [ 1937.355748] LustreError: 64360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852608 with bad export cookie 13490760743300812441 [ 1937.362547] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1937.366376] LustreError: 64360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1937.833588] Lustre: server umount lustre-MDT0001 complete [ 1949.099532] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 1951.867524] Key type lgssc unregistered [ 1952.128421] LNet: 68144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.135481] LNetError: 68144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.153299] LNet: Removed LNI 192.168.202.141@tcp [ 1953.092304] Key type .llcrypt unregistered [ 1953.094769] Key type ._llcrypt unregistered [ 1981.804843] Key type ._llcrypt registered [ 1981.810150] Key type .llcrypt registered [ 1982.010966] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 1983.112125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1983.337782] alg: No test for adler32 (adler32-zlib) [ 1984.513947] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1984.743215] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 1986.385185] Key type lgssc registered [ 1987.964351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1997.663422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1997.674137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.267819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.645871] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2008.567189] LustreError: 69555:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.590318] LustreError: 69555:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2013.678344] LustreError: 69556:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.798637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.566653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2019.003567] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2026.729823] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2032.694307] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2041.167885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2041.853598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.053889] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2060.173521] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2068.656037] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2080.737130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2080.753862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2080.780875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2083.811401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2083.821737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.850671] Lustre: Skipped 1 previous similar message [ 2085.109241] Lustre: server umount lustre-MDT0000 complete [ 2088.948759] LustreError: 69555:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.975273] LustreError: 69555:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2089.924605] LustreError: 69535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852760 with bad export cookie 12201988971595367783 [ 2089.932115] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2089.937873] LustreError: 69535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2090.245135] Lustre: server umount lustre-MDT0001 complete [ 2105.469045] Lustre: server umount lustre-OST0000 complete [ 2139.288143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2139.682623] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2139.721327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2140.043837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.681851] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2156.695644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.740894] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2156.759539] Lustre: 74432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2157.253722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2164.272139] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2175.372776] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2182.350825] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2189.907888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2190.157325] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2190.646748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2198.527043] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2206.318597] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2207.763681] Lustre: 77006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2209.893402] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2217.953670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2217.959770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.989074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2218.002741] Lustre: Skipped 2 previous similar messages [ 2222.403439] Lustre: server umount lustre-OST0000 complete [ 2228.704725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.717057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2228.729043] Lustre: Skipped 1 previous similar message [ 2228.730082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.730087] Lustre: Skipped 2 previous similar messages [ 2228.759973] LustreError: Skipped 1 previous similar message [ 2233.640511] Lustre: server umount lustre-MDT0000 complete [ 2233.826459] LustreError: 74442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.954235] LustreError: 76537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.973731] LustreError: 76537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2239.396837] LustreError: 74422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852910 with bad export cookie 12201988971595368630 [ 2239.407281] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.426663] LustreError: 74422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2240.222818] Lustre: server umount lustre-MDT0001 complete [ 2252.346770] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 2255.698546] Key type lgssc unregistered [ 2256.055552] LNet: 78215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.060311] LNetError: 78215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2256.079786] LNet: Removed LNI 192.168.202.141@tcp [ 2257.101545] Key type .llcrypt unregistered [ 2257.103191] Key type ._llcrypt unregistered [ 2290.145951] Key type ._llcrypt registered [ 2290.147950] Key type .llcrypt registered [ 2290.258648] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2291.124400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2291.133160] alg: No test for adler32 (adler32-zlib) [ 2292.341521] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2292.781563] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2294.535516] Key type lgssc registered [ 2296.060727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2306.644432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2306.693660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.570514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2314.483978] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2318.818653] LustreError: 79629:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.837605] LustreError: 79629:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2323.971083] LustreError: 79630:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.636326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2325.365791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2331.312960] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2340.282112] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2347.413692] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2354.921779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2355.691968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.386575] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2371.445802] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2389.070945] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2390.236759] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2391.483649] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2393.686945] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2393.691772] Lustre: Skipped 1 previous similar message [ 2397.813077] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2397.827888] Lustre: Skipped 3 previous similar messages [ 2406.068565] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2406.076115] Lustre: Skipped 8 previous similar messages [ 2422.716292] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2422.718670] Lustre: Skipped 16 previous similar messages [ 2454.935426] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2454.939643] Lustre: Skipped 33 previous similar messages [ 2502.310386] Lustre: 87761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2505.800605] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 03:39:35 (1788853175) [ 2507.642399] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2510.889623] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 03:39:39 (1788853179) [ 2517.477142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.485155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2517.511576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.623799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.626432] Lustre: Skipped 2 previous similar messages [ 2523.421796] Lustre: server umount lustre-OST0000 complete [ 2529.760307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2529.768635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.780560] Lustre: Skipped 1 previous similar message [ 2529.789677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2532.833275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2532.832685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2532.901063] Lustre: Skipped 1 previous similar message [ 2534.344457] Lustre: server umount lustre-MDT0000 complete [ 2537.960393] LustreError: 80719:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2538.006629] LustreError: 80719:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2539.058886] LustreError: 79611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853209 with bad export cookie 12534540132342701967 [ 2539.062791] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2539.071669] LustreError: 79611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2539.706959] Lustre: server umount lustre-MDT0001 complete [ 2551.220919] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 2554.247184] Key type lgssc unregistered [ 2554.488532] LNet: 89010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2554.496792] LNetError: 89010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2554.519081] LNet: Removed LNI 192.168.202.141@tcp [ 2555.403343] Key type .llcrypt unregistered [ 2555.405331] Key type ._llcrypt unregistered [ 2560.158873] Key type ._llcrypt registered [ 2560.165075] Key type .llcrypt registered [ 2560.344916] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2565.808510] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules [ 2567.078098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2567.115626] alg: No test for adler32 (adler32-zlib) [ 2568.257737] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2568.541703] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2570.271257] Key type lgssc registered [ 2572.222798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2573.534572] LustreError: 90234:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2594.741497] LustreError: 90234:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2597.262661] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2609.602684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2609.634943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.207316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.534769] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2620.466449] LustreError: 91232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.477830] LustreError: 91232:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2625.516486] LustreError: 91233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.624739] LustreError: 91232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.469136] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 03:41:41 (1788853301) [ 2635.747246] LustreError: 91233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.867320] LustreError: 91232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.209622] LustreError: 91264:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2641.235675] LustreError: 91264:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2641.251949] LustreError: 91264:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 31, retries 0, failed: rc = -5 [ 2642.268343] Lustre: server umount lustre-MDT0000 complete [ 2661.909897] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 2664.860495] Key type lgssc unregistered [ 2665.241935] LNet: 92758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2665.246979] LNetError: 92758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2665.264827] LNet: Removed LNI 192.168.202.141@tcp [ 2666.394224] Key type .llcrypt unregistered [ 2666.398708] Key type ._llcrypt unregistered [ 2686.488564] Key type ._llcrypt registered [ 2686.490528] Key type .llcrypt registered [ 2686.582428] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 2687.981884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2688.065182] alg: No test for adler32 (adler32-zlib) [ 2689.099190] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2689.324804] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 2691.015246] Key type lgssc registered [ 2692.670354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2704.120585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2704.158625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2706.140975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2711.746828] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2715.618590] LustreError: 93973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.634849] LustreError: 93973:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2720.738585] LustreError: 93974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.768366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2721.561083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2726.619115] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2733.936965] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2738.730250] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2746.052878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2746.773967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.546817] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2761.424440] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2818.330842] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 03:44:48 (1788853488) [ 2820.537479] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2822.596213] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 03:44:52 (1788853492) [ 2827.747039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2827.750769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2827.769729] Lustre: Skipped 2 previous similar messages [ 2827.782154] Lustre: Skipped 1 previous similar message [ 2832.868389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2832.876230] Lustre: Skipped 3 previous similar messages [ 2833.782993] Lustre: server umount lustre-MDT0000 complete [ 2837.986202] LustreError: 93974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2838.021398] LustreError: 93974:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2838.115483] LustreError: 93955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853508 with bad export cookie 1461296939366795771 [ 2838.142211] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2838.614392] Lustre: server umount lustre-MDT0001 complete [ 2853.049664] Lustre: server umount lustre-OST0000 complete [ 2864.462657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2864.933937] LustreError: 98273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.078811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2869.391567] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2870.242325] LustreError: 98274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2870.260042] LustreError: 98274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2877.712546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2878.557040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2883.796733] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2892.490592] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2898.969691] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2915.193372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2915.449875] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2915.457781] LustreError: 98263:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2915.480933] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2915.499507] LustreError: 100357:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2915.509954] LustreError: 100357:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2915.517571] LustreError: 100357:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2915.600473] Lustre: server umount lustre-OST0000 complete [ 2915.607946] LustreError: 100357:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2927.727689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2927.930044] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2927.937207] LustreError: 98263:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2927.964872] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2927.974683] LustreError: 100824:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2927.988477] LustreError: 100824:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2928.003063] LustreError: 100824:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2928.138881] Lustre: server umount lustre-OST0000 complete [ 2928.141698] LustreError: 100824:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2940.892745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.290739] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2941.299878] Lustre: Found index 0 for lustre-OST0000, updating log [ 2941.329598] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2941.462589] Lustre: lustre-OST0000: new disk, initializing [ 2941.465189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2941.479691] Lustre: 101362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2941.958531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2949.672686] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 2954.894238] Lustre: Failing over lustre-OST0000 [ 2955.392602] Lustre: server umount lustre-OST0000 complete [ 2965.474486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.475812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2965.503365] Lustre: Skipped 1 previous similar message [ 2965.525529] Lustre: Skipped 1 previous similar message [ 2970.601024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.606471] Lustre: Skipped 1 previous similar message [ 2974.687186] Lustre: 93284:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853629/real 1788853629] req@00000000285e4e88 x1875748700921600/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788853645 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2974.687251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.723943] Lustre: 93284:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2974.758748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.765755] Lustre: Skipped 1 previous similar message [ 2975.199126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2975.350417] Lustre: server umount lustre-MDT0000 complete [ 2975.775246] Lustre: 93286:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853630/real 1788853630] req@00000000f1113ed0 x1875748700922496/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788853646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2978.810123] LustreError: 98254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853649 with bad export cookie 1461296939366798515 [ 2978.815697] LustreError: 98254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2978.820559] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2979.063662] Lustre: server umount lustre-MDT0001 complete [ 2987.815163] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 03:47:37 (1788853657) [ 3006.065782] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3018.214948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.827381] LustreError: 103709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.868619] LustreError: 103709:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3019.065828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3025.672815] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3034.488153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.540604] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3044.445436] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3052.270345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3053.220521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.231577] Lustre: Skipped 1 previous similar message [ 3053.256725] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3059.370406] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3059.410367] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3059.414419] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3061.959167] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3074.743618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3074.922143] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3075.105989] Lustre: lustre-OST0001: new disk, initializing [ 3075.112534] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3075.129097] Lustre: 106030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3083.945632] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3085.470058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3085.485315] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3085.674667] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3092.157572] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3103.350995] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3107.297841] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3107.304688] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.317396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3111.408032] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3111.427571] Lustre: Skipped 1 previous similar message [ 3113.769179] Lustre: server umount lustre-MDT0001 complete [ 3116.516375] LustreError: 103710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.547910] LustreError: 103710:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 3124.233308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.375931] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3124.382479] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3124.440397] Lustre: 103699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3125.583566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3125.594038] Lustre: Skipped 1 previous similar message [ 3125.855843] LustreError: 107550:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3125.870515] LustreError: 107550:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3125.882892] LustreError: 103702:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3125.988731] LustreError: 107568:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.141@tcp: cfg command failed: rc = -17 [ 3126.002351] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3126.002351] [ 3130.747480] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3130.865791] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3144.664493] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 03:50:14 (1788853814) [ 3166.959494] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3188.470101] Lustre: 109646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3211.875554] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3215.455608] Lustre: 110781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3303.907781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3303.913989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3303.925863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3303.930942] Lustre: Skipped 4 previous similar messages [ 3308.990417] Lustre: server umount lustre-MDT0000 complete [ 3310.056562] LustreError: 103704:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3310.075723] LustreError: 103704:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 3313.591745] LustreError: 106029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853984 with bad export cookie 1461296939366799516 [ 3313.596778] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3313.608888] LustreError: 106029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3314.106752] Lustre: server umount lustre-MDT0001 complete [ 3329.161494] Lustre: server umount lustre-OST0000 complete [ 3331.040943] Lustre: 93285:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788853985/real 1788853985] req@ffff8eda10f58000 x1875748701194624/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788854001 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3331.070518] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.097633] Lustre: Skipped 3 previous similar messages [ 3332.858705] Lustre: server umount lustre-OST0001 complete [ 3338.430331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3349.615745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3359.502087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3373.033134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3395.555642] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3405.142411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3405.174250] Lustre: lustre-MDT0000: reset Object Index mappings [ 3405.689697] LustreError: 114735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3405.885048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3410.489315] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3419.046818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3419.074778] Lustre: lustre-MDT0001: reset Object Index mappings [ 3425.373278] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3429.041186] Lustre: 115885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3436.460124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3444.306403] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 3445.617624] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3450.389586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3454.838092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3460.606922] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3460.613187] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3462.035754] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3469.863019] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3473.230772] Lustre: 117764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3519.131593] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 03:56:28 (1788854188) [ 3547.637599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3547.645895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3547.679913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3547.693841] Lustre: Skipped 5 previous similar messages [ 3552.743032] LustreError: 115947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3552.757121] LustreError: 115947:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 3552.911651] Lustre: server umount lustre-MDT0000 complete [ 3556.701230] LustreError: 114716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854227 with bad export cookie 1461296939366846388 [ 3556.707105] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3556.708769] LustreError: 114716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3571.167346] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3571.435044] Lustre: server umount lustre-MDT0001 complete [ 3580.789870] Lustre: server umount lustre-OST0000 complete [ 3590.584766] Lustre: server umount lustre-OST0001 complete [ 3597.505342] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 3610.528345] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3652.622936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3652.847551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3652.874655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3652.973810] Lustre: lustre-MDT0000: new disk, initializing [ 3653.119989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3659.171374] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3668.247102] Lustre: server umount lustre-MDT0000 complete [ 3681.652116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.110838] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3682.138759] Lustre: 122042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3682.154930] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3682.424036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3682.431955] Lustre: Skipped 4 previous similar messages [ 3686.682365] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3692.759608] Lustre: server umount lustre-MDT0000 complete [ 3701.472404] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 03:59:31 (1788854371) [ 3703.173345] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3705.027439] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 03:59:35 (1788854375) [ 3722.044136] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 3732.462977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3737.519609] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3749.648785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.858346] Lustre: 124092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3749.969978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3749.977850] Lustre: Skipped 1 previous similar message [ 3750.149197] Lustre: lustre-MDT0001: new disk, initializing [ 3750.332638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3750.359802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3756.556310] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3763.045996] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3770.325099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3770.605500] Lustre: lustre-OST0000: new disk, initializing [ 3770.610944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3770.613992] Lustre: 125670:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3772.683600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3772.703215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3772.867193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3777.838163] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3790.556711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3790.850552] Lustre: lustre-OST0001: new disk, initializing [ 3790.860272] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3790.871490] Lustre: 126553:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3792.867383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3792.878466] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3793.009273] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3798.542822] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 3809.304691] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3813.684560] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4206.842881] Lustre: 132971:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4206.851773] Lustre: 132971:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4206.857725] Lustre: 132971:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4206.865768] Lustre: 132971:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4206.874338] Lustre: 132971:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4206.878488] Lustre: 132971:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4492.894299] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:12:42 (1788855162) [ 4508.129513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4508.139545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4508.154220] Lustre: Skipped 5 previous similar messages [ 4508.158291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4508.161985] Lustre: Skipped 6 previous similar messages [ 4509.983916] Lustre: server umount lustre-MDT0000 complete [ 4514.409385] LustreError: 124083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855185 with bad export cookie 1461296939366912601 [ 4514.421136] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4514.423648] LustreError: 124083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4514.801716] LustreError: 127357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4514.819745] LustreError: 127357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4519.924337] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4519.929583] Lustre: Skipped 6 previous similar messages [ 4521.338692] Lustre: server umount lustre-MDT0001 complete [ 4540.383186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4540.751955] Lustre: server umount lustre-OST0000 complete [ 4551.665167] Lustre: server umount lustre-OST0001 complete [ 4561.312191] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 4574.525687] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 4620.529250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4620.815926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4620.884889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4621.003595] Lustre: lustre-MDT0000: new disk, initializing [ 4621.118440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4621.121153] Lustre: Skipped 4 previous similar messages [ 4621.179307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4626.025965] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4638.246465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4638.361460] Lustre: 136029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4638.407893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4638.414849] Lustre: Skipped 1 previous similar message [ 4638.508467] Lustre: lustre-MDT0001: new disk, initializing [ 4638.723060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4638.743286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4643.699652] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4653.128570] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4659.026916] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4665.956917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4666.315426] Lustre: lustre-OST0000: new disk, initializing [ 4666.320465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4666.326701] Lustre: 137994:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4667.851171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4667.866453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4667.995036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4674.098979] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 4686.310163] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4699.598126] Lustre: lustre-MDD0000: changelog on [ 4702.730582] Lustre: lustre-MDD0001: changelog on [ 6572.636974] Lustre: lustre-MDD0001: changelog off [ 6574.357217] Lustre: 136036:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6575.365902] Lustre: lustre-MDD0000: changelog off [ 6585.832199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6585.877297] Lustre: Skipped 5 previous similar messages [ 6585.891024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6585.897318] Lustre: Skipped 2 previous similar messages [ 6589.637098] Lustre: server umount lustre-OST0000 complete [ 6596.065816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6596.067908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6596.074792] Lustre: Skipped 2 previous similar messages [ 6596.094573] Lustre: Skipped 1 previous similar message [ 6601.184452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6601.801618] Lustre: server umount lustre-MDT0000 complete [ 6605.289550] LustreError: 136039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6605.339801] LustreError: 136039:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6607.158993] LustreError: 136021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857277 with bad export cookie 1461296939367728458 [ 6607.171040] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6607.610547] Lustre: server umount lustre-MDT0001 complete [ 6622.359861] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 6625.542762] Key type lgssc unregistered [ 6625.804273] LNet: 147119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6625.810871] LNetError: 147119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6626.857108] LNet: Removed LNI 192.168.202.141@tcp [ 6628.105670] Key type .llcrypt unregistered [ 6628.108425] Key type ._llcrypt unregistered [ 6655.170244] Key type ._llcrypt registered [ 6655.172061] Key type .llcrypt registered [ 6655.271869] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 6674.221757] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 6675.620534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6675.779982] alg: No test for adler32 (adler32-zlib) [ 6677.096983] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 6677.576343] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 6679.303491] Key type lgssc registered [ 6680.619805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6721.312592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6721.391162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6722.675535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6722.705472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6722.798695] Lustre: lustre-MDT0000: new disk, initializing [ 6722.890417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6722.932771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6727.835989] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6737.440532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.560266] Lustre: 150317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6737.641950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6737.649539] Lustre: Skipped 1 previous similar message [ 6737.781968] Lustre: lustre-MDT0001: new disk, initializing [ 6737.867767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6737.942886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6737.961797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6742.463730] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6753.784954] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6759.335996] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6766.793398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6767.100531] Lustre: lustre-OST0000: new disk, initializing [ 6767.103650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6767.112797] Lustre: 152288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6767.216594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6774.781618] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6775.989387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6775.998864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6776.243912] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6787.764930] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6796.119967] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_module obdecho/obdecho [ 6798.316696] LustreError: 153292:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6798.323676] LustreError: 153292:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6800.225616] LustreError: 153388:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6814.693731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6814.711230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.732116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6817.252478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6817.269625] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.724468] Lustre: server umount lustre-MDT0000 complete [ 6823.308654] LustreError: 150310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857494 with bad export cookie 635781761158504181 [ 6823.323699] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6823.326321] LustreError: 150310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6823.736073] Lustre: server umount lustre-MDT0001 complete [ 6839.438896] Lustre: server umount lustre-OST0000 complete [ 6863.687859] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 6875.161865] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 6912.918349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.156987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6913.179415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6913.234817] Lustre: lustre-MDT0000: new disk, initializing [ 6913.317121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6913.338374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6917.870109] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6929.797328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6929.918848] Lustre: 157107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6929.971359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6929.978149] Lustre: Skipped 1 previous similar message [ 6930.067299] Lustre: lustre-MDT0001: new disk, initializing [ 6930.156979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6930.196340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6930.210242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6936.247486] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6948.605210] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6955.305814] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6963.888302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6964.380742] Lustre: lustre-OST0000: new disk, initializing [ 6964.389345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6964.400117] Lustre: 159078:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6964.528518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6971.719531] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 6972.019346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6972.050262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6972.210950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6981.012536] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6986.033374] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6986.317187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6991.457730] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6991.647598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6997.988413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6997.988927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6998.006842] Lustre: Skipped 1 previous similar message [ 6998.011888] LustreError: Skipped 1 previous similar message [ 6998.014649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6998.014656] Lustre: Skipped 2 previous similar messages [ 7003.137069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7003.142707] Lustre: Skipped 1 previous similar message [ 7008.224816] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7008.229289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7008.247301] Lustre: Skipped 2 previous similar messages [ 7008.471448] Lustre: server umount lustre-OST0000 complete [ 7012.320044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7012.323476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7018.469629] LustreError: 157121:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7018.489172] LustreError: 157121:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7018.679660] Lustre: server umount lustre-MDT0000 complete [ 7022.923085] LustreError: 157101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857693 with bad export cookie 635781761158505140 [ 7022.941458] LustreError: 157101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7022.947077] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7023.353161] Lustre: server umount lustre-MDT0001 complete [ 7028.267396] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 04:54:57 (1788857697) [ 7030.143176] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7032.547472] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 04:55:02 (1788857702) [ 7042.426043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7043.122374] LustreError: 161492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.246046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7048.672780] LustreError: 161493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7050.037995] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7052.775264] LustreError: 161492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.891577] LustreError: 161493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.686986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7065.510720] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7074.698707] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7081.823909] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7090.464345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7090.988471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7090.991821] Lustre: Skipped 1 previous similar message [ 7099.911386] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7111.663632] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7121.380713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7121.406665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7121.418014] Lustre: Skipped 1 previous similar message [ 7121.427442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7121.453618] Lustre: Skipped 3 previous similar messages [ 7124.489542] Lustre: server umount lustre-MDT0000 complete [ 7127.998082] LustreError: 161489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7129.545388] LustreError: 161474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857800 with bad export cookie 635781761158506022 [ 7129.550447] LustreError: 161474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7129.551658] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7133.162591] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7133.166520] Lustre: Skipped 2 previous similar messages [ 7136.027262] Lustre: server umount lustre-MDT0001 complete [ 7144.244584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7160.025133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7160.308214] LustreError: 165336:0:(llog.c:1655:llog_backup()) MGC192.168.202.141@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7160.314162] Lustre: 165336:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.141@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7162.797839] LustreError: 165341:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7162.820309] LustreError: 165341:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 7199.667288] LustreError: 165341:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7199.700880] LustreError: 165341:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 24 previous similar messages [ 7221.224898] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7221.229303] Lustre: 165336:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.141@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7262.230702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7265.202178] LustreError: 165363:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7265.222850] LustreError: 165363:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 46 previous similar messages [ 7267.322477] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7267.889939] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7277.915111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7284.707537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7305.651444] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7305.713255] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7313.634190] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7315.159919] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7321.655503] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7323.451877] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7341.542609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7341.549402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7341.560692] Lustre: Skipped 2 previous similar messages [ 7341.573484] Lustre: Skipped 8 previous similar messages [ 7346.717900] Lustre: server umount lustre-MDT0000 complete [ 7350.703948] LustreError: 161472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858021 with bad export cookie 635781761158512511 [ 7350.708848] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7350.722261] LustreError: 161472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7350.936592] Lustre: server umount lustre-MDT0001 complete [ 7364.449397] Lustre: server umount lustre-OST0000 complete [ 7390.504121] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 7403.526603] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7447.021208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.262306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7447.286609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7447.357086] Lustre: lustre-MDT0000: new disk, initializing [ 7447.420368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7447.432319] Lustre: Skipped 1 previous similar message [ 7447.451343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7452.316623] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7464.449455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7464.579939] Lustre: 170458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7464.828023] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7469.110967] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7480.099763] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7486.020420] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7494.628297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7495.018721] Lustre: lustre-OST0000: new disk, initializing [ 7495.022073] Lustre: Skipped 1 previous similar message [ 7495.033790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7495.043136] Lustre: Skipped 2 previous similar messages [ 7495.054529] Lustre: 172426:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7501.409315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7501.426445] Lustre: Skipped 1 previous similar message [ 7501.438323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7501.583618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7503.104660] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7513.822430] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7518.929528] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7519.150887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7525.037639] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7525.364609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7532.512719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7532.515087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7532.525864] LustreError: Skipped 1 previous similar message [ 7532.555401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7532.577188] Lustre: Skipped 2 previous similar messages [ 7533.895816] Lustre: server umount lustre-OST0000 complete [ 7544.968960] Lustre: server umount lustre-MDT0000 complete [ 7547.873939] LustreError: 170469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.900797] LustreError: 170469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 35 previous similar messages [ 7549.710258] LustreError: 170450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858220 with bad export cookie 635781761158529311 [ 7549.720628] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7549.724516] LustreError: 170450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7550.307889] Lustre: server umount lustre-MDT0001 complete [ 7559.698559] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 05:03:49 (1788858229) [ 7567.578535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.430819] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7583.824982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.935345] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7599.884878] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7606.571127] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7613.159233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7620.707504] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7628.921791] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7650.627589] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 05:05:20 (1788858320) [ 7658.468857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7658.478655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7658.491364] Lustre: Skipped 4 previous similar messages [ 7658.499057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7658.499065] Lustre: Skipped 3 previous similar messages [ 7658.506496] LustreError: Skipped 1 previous similar message [ 7661.458813] Lustre: server umount lustre-OST0000 complete [ 7672.438474] Lustre: server umount lustre-MDT0000 complete [ 7678.008957] LustreError: 174821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858348 with bad export cookie 635781761158530193 [ 7678.019602] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7678.368747] Lustre: server umount lustre-MDT0001 complete [ 7691.476539] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 7694.364168] Key type lgssc unregistered [ 7694.710123] LNet: 178594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7694.720314] LNetError: 178594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7694.752336] LNet: Removed LNI 192.168.202.141@tcp [ 7695.776062] Key type .llcrypt unregistered [ 7695.778500] Key type ._llcrypt unregistered [ 7716.058738] Key type ._llcrypt registered [ 7716.061310] Key type .llcrypt registered [ 7716.159503] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7717.171163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7717.293772] alg: No test for adler32 (adler32-zlib) [ 7718.376949] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 7718.626344] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 7720.295182] Key type lgssc registered [ 7721.324691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7728.801519] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7728.816577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7744.352780] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7749.473571] LustreError: 179833:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.141@tcp: failed processing log, type 4: rc = -110 [ 7754.659740] LustreError: 179833:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7754.680385] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7754.699549] LustreError: MGC192.168.202.141@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7754.725504] LustreError: 179833:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7754.733670] LustreError: 179833:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7754.751181] LustreError: 179833:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7754.843584] Lustre: server umount lustre-OST0000 complete [ 7754.847559] LustreError: 179833:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7765.085131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7766.545843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7772.267162] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7776.745486] LustreError: 180333:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7776.792036] LustreError: 180333:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7781.871528] LustreError: 180334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7782.574603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7783.261020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7787.849370] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7795.993203] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7802.764390] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7809.525878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7809.690893] LustreError: 182150:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7809.700099] Lustre: 182150:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.141@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7809.832394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7816.354061] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7823.755355] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7827.427383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7827.435785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7827.462480] Lustre: Skipped 2 previous similar messages [ 7827.485890] Lustre: Skipped 2 previous similar messages [ 7828.514423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7828.525019] Lustre: Skipped 2 previous similar messages [ 7832.399448] Lustre: server umount lustre-MDT0000 complete [ 7833.570872] LustreError: 180330:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7833.596611] LustreError: 180330:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7836.364438] LustreError: 180314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858507 with bad export cookie 9659519225372974052 [ 7836.374595] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7836.727291] LustreError: 181063:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004ba80b1b x1875753974134400/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7836.976262] Lustre: server umount lustre-MDT0001 complete [ 7851.552237] Lustre: server umount lustre-OST0000 complete [ 7865.831241] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 7878.682484] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 7916.771687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7917.123994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7917.164305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7917.283116] Lustre: lustre-MDT0000: new disk, initializing [ 7917.411074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7917.436490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7921.879333] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7933.615514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7933.760968] Lustre: 185676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7933.824820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7933.831925] Lustre: Skipped 1 previous similar message [ 7933.914381] Lustre: lustre-MDT0001: new disk, initializing [ 7934.058774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7934.102782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7934.118310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7939.264089] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7948.683617] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7954.889376] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7962.494537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7962.778899] Lustre: lustre-OST0000: new disk, initializing [ 7962.785847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7962.796538] Lustre: 187643:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7962.944323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7969.348803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7969.363189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7969.493828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7969.563449] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 7979.660920] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7984.562875] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7984.862243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7990.199230] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7990.535386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7995.364963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7995.369940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7995.394630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7995.418619] Lustre: Skipped 1 previous similar message [ 8000.503981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8000.514409] Lustre: Skipped 1 previous similar message [ 8005.614060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8005.621591] Lustre: Skipped 1 previous similar message [ 8007.135301] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8007.369432] Lustre: server umount lustre-OST0000 complete [ 8015.848949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8015.861055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8015.868814] Lustre: Skipped 2 previous similar messages [ 8015.869993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8015.897040] LustreError: Skipped 1 previous similar message [ 8017.370412] Lustre: server umount lustre-MDT0000 complete [ 8020.961866] LustreError: 185690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8020.982311] LustreError: 185690:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8021.492836] LustreError: 185669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858692 with bad export cookie 9659519225372974773 [ 8021.522962] LustreError: 185669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8021.537300] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8021.819938] Lustre: server umount lustre-MDT0001 complete [ 8031.367130] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 05:11:41 (1788858701) [ 8050.373920] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 8053.548378] Key type lgssc unregistered [ 8053.897669] LNet: 190592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8053.910860] LNetError: 190592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8053.929275] LNet: Removed LNI 192.168.202.141@tcp [ 8054.994946] Key type .llcrypt unregistered [ 8055.000196] Key type ._llcrypt unregistered [ 8075.947354] Key type ._llcrypt registered [ 8075.949520] Key type .llcrypt registered [ 8076.069343] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8077.719527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8077.820218] alg: No test for adler32 (adler32-zlib) [ 8078.907465] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 8079.286209] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 8081.063220] Key type lgssc registered [ 8082.363292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8090.451855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8090.468881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8091.928777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8097.589168] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8101.363128] LustreError: 191808:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8101.380390] LustreError: 191808:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8106.474525] LustreError: 191809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8107.985232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8108.398905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8113.244254] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8121.560305] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8127.480194] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8135.452144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8135.674860] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8135.684157] LustreError: 193671:0:(llog.c:1696:llog_backup()) MGC192.168.202.141@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8135.692049] Lustre: 193671:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.141@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8135.831883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8142.729804] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8151.472770] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8154.597083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8154.608450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.624453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8155.686160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8155.690367] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8155.723244] Lustre: Skipped 1 previous similar message [ 8160.308045] Lustre: server umount lustre-MDT0000 complete [ 8160.741643] LustreError: 191805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8160.759135] LustreError: 191805:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8164.643540] LustreError: 191789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858835 with bad export cookie 16970835848060055195 [ 8164.646901] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8164.652865] LustreError: 191789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8165.856199] LustreError: 194431:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8165.861110] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8165.905209] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8165.915634] Lustre: Skipped 2 previous similar messages [ 8165.984741] LustreError: 192538:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a6c2ce54 x1875754352670080/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8166.180270] Lustre: server umount lustre-MDT0001 complete [ 8172.130291] Lustre: server umount lustre-OST0000 complete [ 8182.122928] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 8192.297780] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8228.984349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.296612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8229.328148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8229.458171] Lustre: lustre-MDT0000: new disk, initializing [ 8229.544777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8229.582534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8234.764733] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8246.282956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.440758] Lustre: 197193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8246.482763] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8246.484669] Lustre: Skipped 1 previous similar message [ 8246.487329] LustreError: 197977:0:(llog.c:1696:llog_backup()) MGC192.168.202.141@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8246.494449] LustreError: 197977:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8246.499227] Lustre: 197977:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.141@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8246.522427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8246.531608] Lustre: Skipped 1 previous similar message [ 8246.607694] Lustre: lustre-MDT0001: new disk, initializing [ 8246.713614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8246.770914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8246.781589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8252.110985] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8264.040783] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8270.832396] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8278.259186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8278.606739] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8278.609795] Lustre: Skipped 1 previous similar message [ 8278.613844] LustreError: 199139:0:(llog.c:1696:llog_backup()) MGC192.168.202.141@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8278.624660] LustreError: 199139:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8278.632293] Lustre: 199139:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.141@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8278.640927] Lustre: 199139:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8278.663924] Lustre: lustre-OST0000: new disk, initializing [ 8278.667712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8278.678497] Lustre: 199163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8278.782848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8283.819954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8283.835582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8283.934154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8285.752358] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8296.838819] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8302.071102] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8302.355550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8308.295674] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8308.474905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8314.857583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8314.859159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8314.873270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8314.901940] Lustre: Skipped 1 previous similar message [ 8316.965735] Lustre: server umount lustre-OST0000 complete [ 8323.552374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8323.571738] LustreError: Skipped 1 previous similar message [ 8323.577816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8323.595274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8323.602425] Lustre: Skipped 1 previous similar message [ 8334.309048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8334.315173] Lustre: Skipped 5 previous similar messages [ 8335.327155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8335.539434] Lustre: server umount lustre-MDT0000 complete [ 8339.426968] LustreError: 197205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8339.455518] LustreError: 197205:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8340.695528] LustreError: 197185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859011 with bad export cookie 16970835848060055916 [ 8340.709526] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8340.712583] LustreError: 197185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8341.262814] Lustre: server umount lustre-MDT0001 complete [ 8352.402223] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 05:17:01 (1788859021) [ 8370.707803] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8382.720604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8383.230298] LustreError: 202295:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8383.262445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8388.311421] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8397.320516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8402.465842] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8405.366413] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8413.318282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8420.603425] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8430.625103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8430.887361] Lustre: lustre-OST0001: new disk, initializing [ 8430.892737] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8430.900284] Lustre: 204603:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8431.030916] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8431.039380] Lustre: Skipped 2 previous similar messages [ 8440.008114] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8440.550344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8440.561152] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8440.723043] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8452.190671] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8463.173575] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8465.274959] Lustre: DEBUG MARKER: ADD OST3 [ 8470.959943] loop: module loaded [ 8471.087134] loop0: detected capacity change from 0 to 204800000 [ 8471.397690] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8471.421088] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8471.437328] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8478.329816] loop0: detected capacity change from 0 to 204800000 [ 8479.096528] Lustre: DEBUG MARKER: STOP OST3 [ 8486.639344] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8487.104423] Lustre: lustre-OST0003: new disk, initializing [ 8487.115491] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8487.139069] Lustre: 206637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8488.873971] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8489.043133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8489.073517] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8489.119695] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8490.912285] LustreError: 206762:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8491.299609] Lustre: Failing over lustre-OST0003 [ 8491.782364] Lustre: server umount lustre-OST0003 complete [ 8494.034151] LustreError: 203780:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8494.059164] LustreError: 203780:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8494.098553] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8494.101793] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8494.122240] Lustre: Skipped 1 previous similar message [ 8495.830501] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8504.504351] Lustre: DEBUG MARKER: START OST3 again [ 8512.857954] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8513.461787] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8513.470417] Lustre: Skipped 1 previous similar message [ 8513.495772] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8514.482262] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8514.926319] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8514.926643] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8520.885256] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8544.216661] Lustre: Failing over lustre-OST0003 [ 8544.465858] Lustre: server umount lustre-OST0003 complete [ 8545.761463] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8545.767043] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8545.785060] LustreError: Skipped 1 previous similar message [ 8545.819849] LustreError: 203781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8545.820333] Lustre: Skipped 2 previous similar messages [ 8545.873662] LustreError: 203781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 8566.753634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8566.771845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8571.341976] Lustre: server umount lustre-MDT0000 complete [ 8576.962106] LustreError: 202275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859247 with bad export cookie 16970835848060056798 [ 8576.973359] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8576.982548] LustreError: 202275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8578.018072] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8578.041781] Lustre: Skipped 5 previous similar messages [ 8591.327661] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8591.959756] Lustre: server umount lustre-MDT0001 complete [ 8597.415810] Lustre: server umount lustre-OST0000 complete [ 8602.219461] Lustre: server umount lustre-OST0001 complete [ 8610.296660] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 8623.544865] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8668.623017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8668.978819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8669.003296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8669.061331] Lustre: lustre-MDT0000: new disk, initializing [ 8669.138110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8669.176275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8675.112787] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8686.587851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8686.752580] Lustre: 212070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8686.828455] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8686.834945] Lustre: Skipped 1 previous similar message [ 8686.922629] Lustre: lustre-MDT0001: new disk, initializing [ 8687.129355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8687.143829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8691.549381] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8701.979590] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8707.338569] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8716.595475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8717.025855] Lustre: 214040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8718.511324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8718.663934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8724.888384] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8737.225987] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8744.395722] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8744.763386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8750.342532] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8750.659599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8754.660859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8754.662896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8754.665768] LustreError: Skipped 1 previous similar message [ 8754.674634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8754.678925] Lustre: Skipped 1 previous similar message [ 8754.696890] Lustre: Skipped 13 previous similar messages [ 8758.984887] Lustre: server umount lustre-OST0000 complete [ 8769.771992] Lustre: server umount lustre-MDT0000 complete [ 8770.021752] LustreError: 213547:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8770.059987] LustreError: 213547:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 23 previous similar messages [ 8775.818485] LustreError: 212874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859446 with bad export cookie 16970835848060059094 [ 8775.838658] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8776.443315] Lustre: server umount lustre-MDT0001 complete [ 8781.540025] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 05:24:11 (1788859451) [ 8801.105821] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 8813.583433] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 8852.493203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8852.779788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8852.815269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8852.818737] Lustre: Skipped 1 previous similar message [ 8852.898264] Lustre: lustre-MDT0000: new disk, initializing [ 8852.900116] Lustre: Skipped 1 previous similar message [ 8853.031626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8853.044477] Lustre: Skipped 1 previous similar message [ 8858.309309] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8869.944506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8870.129311] Lustre: 218653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8870.380663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8876.664375] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8887.456665] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8893.369720] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8900.609292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8900.825930] Lustre: 220619:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8902.671442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8902.839590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8907.385531] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8920.689626] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8927.891326] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8928.206887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8934.115711] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8934.344542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8938.466830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8938.476293] LustreError: Skipped 1 previous similar message [ 8938.479866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8938.491157] Lustre: Skipped 2 previous similar messages [ 8938.503627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8938.516541] Lustre: Skipped 3 previous similar messages [ 8943.445900] Lustre: server umount lustre-OST0000 complete [ 8954.600888] Lustre: server umount lustre-MDT0000 complete [ 8958.440960] LustreError: 218660:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8958.451327] LustreError: 218660:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8958.798151] LustreError: 220618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859629 with bad export cookie 16970835848060059976 [ 8958.813945] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8958.819106] LustreError: 220618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8959.132700] Lustre: server umount lustre-MDT0001 complete [ 8967.799325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8968.397097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8968.400137] Lustre: Skipped 5 previous similar messages [ 8974.134609] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8983.939208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8990.340582] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 8999.225549] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9005.800894] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9013.134574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9020.127642] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9029.486842] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9042.926615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 9042.934027] LustreError: Skipped 1 previous similar message [ 9047.933544] Lustre: server umount lustre-OST0000 complete [ 9059.444807] Lustre: server umount lustre-MDT0000 complete [ 9064.122160] LustreError: 222820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788859734 with bad export cookie 16970835848060060858 [ 9064.146318] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9064.170265] LustreError: 222820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9064.897227] Lustre: server umount lustre-MDT0001 complete [ 9170.214366] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [ 9174.956393] Key type lgssc unregistered [ 9175.451335] LNet: 226508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9175.463813] LNetError: 226508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9176.490484] LNet: Removed LNI 192.168.202.141@tcp [ 9177.668171] Key type .llcrypt unregistered [ 9177.670869] Key type ._llcrypt unregistered [ 9208.863707] Key type ._llcrypt registered [ 9208.871384] Key type .llcrypt registered [ 9208.999993] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 9224.927106] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9226.203982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9226.316845] alg: No test for adler32 (adler32-zlib) [ 9227.400710] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 9227.766279] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [ 9229.455210] Key type lgssc registered [ 9231.365467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9276.974880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9277.003119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9278.436087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9278.503024] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9278.612423] Lustre: lustre-MDT0000: new disk, initializing [ 9278.746570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9278.775702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9283.946603] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9297.557770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9297.621526] Lustre: 229863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9297.656888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9297.660300] Lustre: Skipped 1 previous similar message [ 9297.709756] Lustre: lustre-MDT0001: new disk, initializing [ 9297.814623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9297.841651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9297.856189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9303.801840] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9315.632982] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9322.034503] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9329.011443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9329.365468] Lustre: lustre-OST0000: new disk, initializing [ 9329.369734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9329.390367] Lustre: 231829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9329.533247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9336.624917] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9336.935834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9336.948048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9337.066946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9346.989365] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9352.825130] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9353.095197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9357.931624] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9358.144466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9362.917721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9362.920452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9362.928390] LustreError: Skipped 1 previous similar message [ 9362.930742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9362.985684] Lustre: Skipped 1 previous similar message [ 9368.033906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9368.043495] Lustre: Skipped 2 previous similar messages [ 9373.170375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9373.188197] Lustre: Skipped 1 previous similar message [ 9374.175157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9374.600750] Lustre: server umount lustre-OST0000 complete [ 9379.808749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9379.826371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9379.851264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9383.394867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9388.515355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9388.526090] Lustre: Skipped 3 previous similar messages [ 9393.119182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9393.330459] Lustre: server umount lustre-MDT0000 complete [ 9397.418344] LustreError: 229854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860068 with bad export cookie 16976705869636687762 [ 9397.425995] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9397.436318] LustreError: 229854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9398.025859] Lustre: server umount lustre-MDT0001 complete [ 9418.869141] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9429.529705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9429.869201] LustreError: 234766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9429.890183] LustreError: 234766:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9429.954522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9434.460940] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9435.105259] LustreError: 234767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9440.225785] LustreError: 234766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9443.418535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9443.843446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9448.355722] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9451.426643] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9459.225988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9459.675448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9466.830951] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9476.102962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9476.338604] Lustre: lustre-OST0001: new disk, initializing [ 9476.350088] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9476.357302] Lustre: 237074:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9476.486069] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9483.370823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9483.381622] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9483.550613] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9484.427046] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9497.758145] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9506.031411] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9519.443365] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 05:36:29 (1788860189) [ 9524.709724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9524.714160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9524.744991] Lustre: Skipped 3 previous similar messages [ 9524.778847] Lustre: Skipped 3 previous similar messages [ 9528.398289] Lustre: server umount lustre-MDT0000 complete [ 9529.826248] LustreError: 234767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9529.870085] LustreError: 234767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9533.878461] LustreError: 237073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860204 with bad export cookie 16976705869636688644 [ 9533.888903] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9533.900365] LustreError: 237073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9534.477845] Lustre: server umount lustre-MDT0001 complete [ 9549.820300] Lustre: server umount lustre-OST0000 complete [ 9550.316908] Lustre: 228085:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788860205/real 1788860205] req@ffff8ed906ec5c00 x1875755556524160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788860221 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9550.366857] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9554.966382] Lustre: server umount lustre-OST0001 complete [ 9564.330788] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [ 9578.412455] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [ 9624.299924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9624.586310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9624.626101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9624.695386] Lustre: lustre-MDT0000: new disk, initializing [ 9624.770615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9624.793520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9630.138668] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9641.706733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9641.862967] Lustre: 241022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9641.912916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9641.915837] Lustre: Skipped 1 previous similar message [ 9641.969474] Lustre: lustre-MDT0001: new disk, initializing [ 9642.167755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9642.181849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9646.792971] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9656.730584] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9662.930706] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9670.373328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9670.665640] Lustre: lustre-OST0000: new disk, initializing [ 9670.669612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9670.678292] Lustre: 242990:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9672.131181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9672.147514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9672.258170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9677.862893] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9687.998736] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9693.582435] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9693.897476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9698.891511] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9699.194287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9702.880862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9702.890465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9702.898790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9702.915917] Lustre: Skipped 1 previous similar message [ 9707.802859] Lustre: server umount lustre-OST0000 complete [ 9719.442267] Lustre: server umount lustre-MDT0000 complete [ 9722.359691] LustreError: 241027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9722.388084] LustreError: 241027:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9725.155554] LustreError: 242989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860395 with bad export cookie 16976705869636690779 [ 9725.174217] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9725.174697] LustreError: 242989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9725.963204] Lustre: server umount lustre-MDT0001 complete [ 9736.947521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9737.593187] LustreError: 245208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9737.700843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9737.706246] Lustre: Skipped 2 previous similar messages [ 9743.020378] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9753.112271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9759.597597] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9769.939047] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9777.528934] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9786.773656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9792.919213] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [ 9801.315508] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9951.200655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9951.207845] LustreError: Skipped 1 previous similar message [ 9951.211053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9951.221528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9951.242276] Lustre: Skipped 3 previous similar messages [ 9951.246171] Lustre: Skipped 5 previous similar messages [ 9955.617594] Lustre: server umount lustre-OST0000 complete [ 9968.596982] Lustre: server umount lustre-MDT0000 complete [ 9971.682060] LustreError: 245209:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9971.696981] LustreError: 245209:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9973.206373] LustreError: 245188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860643 with bad export cookie 16976705869636691654 [ 9973.209592] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9973.223500] LustreError: 245188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9973.950267] Lustre: server umount lustre-MDT0001 complete [10073.316413] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [10077.454197] Key type lgssc unregistered [10077.718396] LNet: 248812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10077.724782] LNetError: 248812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10077.739456] LNet: Removed LNI 192.168.202.141@tcp [10078.663481] Key type .llcrypt unregistered [10078.670515] Key type ._llcrypt unregistered [10110.457542] Key type ._llcrypt registered [10110.460646] Key type .llcrypt registered [10110.558534] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [10126.579877] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10127.916708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10128.251688] alg: No test for adler32 (adler32-zlib) [10129.322964] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [10129.540190] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [10131.283430] Key type lgssc registered [10133.001404] Lustre: Echo OBD driver; http://www.lustre.org/ [10180.640679] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10191.298387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10191.332518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10192.688194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10192.729431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10192.830852] Lustre: lustre-MDT0000: new disk, initializing [10192.935413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10192.962848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10197.907393] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10209.408824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10209.580625] Lustre: 252880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10209.701069] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10209.705544] Lustre: Skipped 1 previous similar message [10209.784487] Lustre: lustre-MDT0001: new disk, initializing [10209.972909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10210.012607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10210.031047] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10215.034911] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10219.718490] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10227.830446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10228.392288] Lustre: lustre-OST0000: new disk, initializing [10228.404026] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10228.412258] Lustre: 254512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10228.563449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10234.930664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10234.939305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10235.057381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10235.592034] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10247.139946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10247.339345] Lustre: lustre-OST0001: new disk, initializing [10247.342572] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10247.348566] Lustre: 255382:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10247.429590] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10252.883692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10252.896542] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10252.970521] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10254.021690] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10265.553899] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10276.279970] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10292.193126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10292.198755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10292.210389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10294.244526] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10294.246266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10294.260144] Lustre: Skipped 1 previous similar message [10294.284551] Lustre: Skipped 3 previous similar messages [10297.471656] Lustre: server umount lustre-MDT0000 complete [10301.736759] LustreError: 252874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788860972 with bad export cookie 9410580401519400696 [10301.737199] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10301.756434] LustreError: 252874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10302.131890] Lustre: server umount lustre-MDT0001 complete [10316.882559] Lustre: server umount lustre-OST0000 complete [10320.866512] Lustre: 250388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788860975/real 1788860975] req@ffff8eda2d25b100 x1875756502302976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788860991 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10320.930679] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.983633] Lustre: Skipped 1 previous similar message [10322.915360] Lustre: 250387:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788860977/real 1788860977] req@ffff8eda36894700 x1875756502303232/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788860993 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10324.262828] Lustre: server umount lustre-OST0001 complete [10343.287791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10356.618521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10367.478242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10381.412198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10410.950795] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10425.639956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.662916] Lustre: lustre-MDT0000: reset Object Index mappings [10426.453868] LustreError: 260400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10426.498581] LustreError: 260400:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10426.587538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10431.971777] LustreError: 260401:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10432.914976] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10437.089809] LustreError: 260400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10442.220285] LustreError: 260401:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10444.385520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10444.405605] Lustre: lustre-MDT0001: reset Object Index mappings [10445.124721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10451.090638] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10454.918647] Lustre: 261544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10463.363849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10463.994348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10470.138627] LustreError: 261897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10471.553079] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10472.250470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10482.678622] LustreError: 261898:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10482.705959] LustreError: 261898:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10483.128673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10488.879666] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10493.358839] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10502.503868] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10507.374523] Lustre: 263415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10514.406685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10514.411599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10514.427040] Lustre: Skipped 3 previous similar messages [10514.445613] Lustre: Skipped 3 previous similar messages [10515.505491] Lustre: server umount lustre-MDT0000 complete [10519.527284] LustreError: 261351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10519.549411] LustreError: 261351:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10519.865332] LustreError: 260381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861190 with bad export cookie 9410580401519407486 [10519.893297] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10520.313312] Lustre: server umount lustre-MDT0001 complete [10535.184772] Lustre: server umount lustre-OST0000 complete [10550.266528] Lustre: server umount lustre-OST0001 complete [10574.852067] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 05:54:04 (1788861244) [10594.882984] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [10609.709901] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10653.868636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10654.135675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10654.161742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10654.269835] Lustre: lustre-MDT0000: new disk, initializing [10654.421050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10654.433354] Lustre: Skipped 1 previous similar message [10654.493132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10660.517962] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10673.372891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10673.527755] Lustre: 267673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10673.593862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10673.601792] Lustre: Skipped 1 previous similar message [10673.752767] Lustre: lustre-MDT0001: new disk, initializing [10673.995190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10674.014777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10681.153173] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10694.428654] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10701.844324] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10709.858676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10710.352987] Lustre: lustre-OST0000: new disk, initializing [10710.356524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10710.363819] Lustre: 269642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10711.801048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10711.820491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10711.989899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10716.617654] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10727.723810] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10733.165832] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10733.423282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10738.562379] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10738.865428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10742.763894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10742.768352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10742.768435] LustreError: Skipped 1 previous similar message [10742.772795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10742.778878] Lustre: Skipped 1 previous similar message [10747.576430] Lustre: server umount lustre-OST0000 complete [10752.803801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10752.819261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10752.842389] Lustre: Skipped 1 previous similar message [10752.869366] Lustre: Skipped 2 previous similar messages [10763.235081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10763.245521] Lustre: Skipped 3 previous similar messages [10767.327906] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10767.761884] Lustre: server umount lustre-MDT0000 complete [10768.354411] LustreError: 267685:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10768.386200] LustreError: 267685:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10772.386198] LustreError: 269641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861443 with bad export cookie 9410580401519409579 [10772.400264] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10772.407785] LustreError: 269641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10772.946356] Lustre: server umount lustre-MDT0001 complete [10797.353789] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10809.138443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10809.737383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10809.748649] Lustre: Skipped 2 previous similar messages [10813.783720] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10824.979718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10830.937663] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10834.264731] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10841.255571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10848.808383] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10857.940298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10858.169480] Lustre: lustre-OST0001: new disk, initializing [10858.177921] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10858.192743] Lustre: 274887:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10859.742382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10859.756709] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10859.917608] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10866.133582] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10878.700122] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10884.037131] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10891.097471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10891.181958] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10906.101664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10906.107269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10906.129760] Lustre: Skipped 3 previous similar messages [10906.146519] Lustre: Skipped 3 previous similar messages [10909.452326] Lustre: server umount lustre-MDT0000 complete [10911.200854] LustreError: 273320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10911.241340] LustreError: 273320:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10913.468143] LustreError: 272559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861584 with bad export cookie 9410580401519410461 [10913.483613] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10913.488235] LustreError: 272559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10914.005146] Lustre: server umount lustre-MDT0001 complete [10927.545249] Lustre: server umount lustre-OST0000 complete [10941.656709] Lustre: server umount lustre-OST0001 complete [10960.935242] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [10972.002821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10972.684533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10972.704720] Lustre: Skipped 3 previous similar messages [10979.155502] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [10991.370428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10998.929120] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11002.465831] Lustre: 279386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11009.954262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11018.694618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11018.695548] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11029.147904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11034.616390] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11036.356985] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11045.777575] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11050.143797] Lustre: 281265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11056.028943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11056.040106] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11070.286143] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 06:02:20 (1788861740) [11089.376640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11089.384137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11089.399257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11092.973711] Lustre: server umount lustre-MDT0000 complete [11096.042258] LustreError: 278234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11096.075835] LustreError: 278234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [11097.143593] LustreError: 279764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861767 with bad export cookie 9410580401519417307 [11097.147592] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11097.156496] LustreError: 279764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11103.852351] Lustre: server umount lustre-MDT0001 complete [11114.253496] Lustre: server umount lustre-OST0000 complete [11118.551888] Lustre: server umount lustre-OST0001 complete [11126.437370] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [11138.438923] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11186.854604] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11197.422263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11197.830256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11197.874078] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11197.992344] Lustre: lustre-MDT0000: new disk, initializing [11198.122296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11203.771887] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11215.888948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11216.082385] Lustre: 285796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11216.729317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11221.833821] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11235.853377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11236.231275] Lustre: 287428:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11236.369650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11236.386066] Lustre: Skipped 5 previous similar messages [11238.352753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11244.932589] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11259.488384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11259.867975] Lustre: 288299:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11261.514973] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11261.535279] Lustre: Skipped 1 previous similar message [11261.667527] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11267.428669] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11278.494329] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11283.330499] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11283.336841] Lustre: Skipped 1 previous similar message [11305.127264] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 06:06:14 (1788861974) [11308.033266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11308.044518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11308.052682] Lustre: Skipped 6 previous similar messages [11308.085189] Lustre: Skipped 10 previous similar messages [11322.335651] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11322.742787] Lustre: server umount lustre-MDT0000 complete [11328.042910] LustreError: 286588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788861998 with bad export cookie 9410580401519424790 [11328.049935] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11328.060310] LustreError: 286588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11328.507548] Lustre: server umount lustre-MDT0001 complete [11343.864408] Lustre: server umount lustre-OST0000 complete [11344.863826] Lustre: 250388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788861999/real 1788861999] req@ffff8eda3dd71880 x1875756502646144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788862015 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11348.402643] Lustre: server umount lustre-OST0001 complete [11359.617430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11360.524946] LustreError: 290549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11360.548719] LustreError: 290549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [11366.269506] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11376.212392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11382.068900] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11389.823067] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11396.015435] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11403.190769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11411.125972] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11419.636873] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11442.955670] loop1: detected capacity change from 0 to 204800000 [11443.366995] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11443.381426] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11443.399727] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11449.450127] loop1: detected capacity change from 0 to 204800000 [11456.093346] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11456.248783] LustreError: 290539:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs3636: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11456.269866] LustreError: 290539:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs3636 [11456.276564] LustreError: 290539:0:(mgs_handler.c:517:mgs_target_reg()) fs3636-MDT0001: Error checking target index: rc = -13 [11456.288777] LustreError: fs3636-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11456.301336] LustreError: 294271:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11456.307840] LustreError: 294271:0:(tgt_mount.c:2128:server_put_super()) no obd fs3636-MDT0001 [11456.312547] LustreError: 294271:0:(tgt_mount.c:116:server_deregister_mount()) fs3636-MDT0001 not registered [11456.361831] Lustre: server umount fs3636-MDT0001 complete [11456.364799] LustreError: 294271:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11465.493161] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11465.631658] Lustre: Setting parameter fs3636-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs3636-MDT0001 [11465.754852] Lustre: srv-fs3636-MDT0001: No data found on store. Initialize space. [11465.771657] Lustre: Skipped 4 previous similar messages [11465.881993] Lustre: fs3636-MDT0001: new disk, initializing [11465.886404] Lustre: Skipped 3 previous similar messages [11466.020416] Lustre: srv-fs3636-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11466.028185] Lustre: 294737:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3636-MDT0001-mdtlov: can't create new llog: rc = -115 [11466.038206] LustreError: 294737:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3636-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11471.230484] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11478.482377] Lustre: server umount fs3636-MDT0001 complete [11483.630543] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11493.647662] loop1: detected capacity change from 0 to 204800000 [11499.457796] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11499.779639] Lustre: 296307:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3636-MDT0001-mdtlov: can't create new llog: rc = -115 [11499.795250] LustreError: 296307:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3636-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11504.108799] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11504.587494] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11509.928811] Lustre: server umount fs3636-MDT0001 complete [11523.532566] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [11523.541700] Lustre: Skipped 8 previous similar messages [11524.589320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11524.604832] Lustre: Skipped 5 previous similar messages [11535.327156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11535.673553] Lustre: server umount lustre-MDT0000 complete [11544.645946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11544.775726] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11550.184246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11550.769299] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11559.457769] loop1: detected capacity change from 0 to 204800000 [11566.818294] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11567.482581] Lustre: 298755:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3636-MDT0001-mdtlov: can't create new llog: rc = -115 [11567.499063] LustreError: 298755:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3636-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11572.988819] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11578.696403] Lustre: server umount fs3636-MDT0001 complete [11611.642068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11623.391146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11623.562554] Lustre: server umount lustre-OST0000 complete [11642.335231] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11642.654465] Lustre: server umount lustre-MDT0000 complete [11647.471693] LustreError: 290530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862318 with bad export cookie 9410580401519427471 [11647.476138] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11647.482511] LustreError: 290530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11648.044881] Lustre: server umount lustre-MDT0001 complete [11660.526682] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [11663.983977] Key type lgssc unregistered [11664.381714] LNet: 301102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11664.392342] LNetError: 301102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11664.417243] LNet: Removed LNI 192.168.202.141@tcp [11665.363230] Key type .llcrypt unregistered [11665.365656] Key type ._llcrypt unregistered [11687.316554] Key type ._llcrypt registered [11687.319387] Key type .llcrypt registered [11687.446335] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [11688.929342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11689.030470] alg: No test for adler32 (adler32-zlib) [11690.049546] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [11690.260155] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [11691.951349] Key type lgssc registered [11693.371181] Lustre: Echo OBD driver; http://www.lustre.org/ [11702.418904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11702.452844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11704.026242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11709.075500] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11713.509074] LustreError: 302374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11713.520578] LustreError: 302374:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11718.631663] LustreError: 302375:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11718.860405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11719.557318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11726.081896] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11735.427256] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11743.314369] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11752.008147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11752.390546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11759.994334] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11764.748739] LustreError: 304205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11769.835205] LustreError: 304208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11770.713539] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11773.926560] LustreError: 304208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11773.938753] LustreError: 304208:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11782.584776] LustreError: 304879:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11782.605156] LustreError: 304879:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11796.399138] loop1: detected capacity change from 0 to 204800000 [11796.663715] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11796.679512] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11796.706114] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11799.546154] LustreError: 304207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11799.570900] LustreError: 304207:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [11802.773663] loop1: detected capacity change from 0 to 204800000 [11810.215503] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11810.477369] Lustre: fs22097-OST0000: new disk, initializing [11810.494623] Lustre: srv-fs22097-OST0000: No data found on store. Initialize space. [11810.500943] Lustre: 306113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11810.644748] Lustre: fs22097-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11818.652577] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11828.341465] Lustre: server umount fs22097-OST0000 complete [11833.790127] LustreError: 304879:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11833.833188] LustreError: 304879:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [11843.050460] loop1: detected capacity change from 0 to 204800000 [11845.614803] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11850.761734] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11850.972792] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11851.130863] Lustre: fs22097-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11859.172859] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11869.476065] loop2: detected capacity change from 0 to 204800000 [11869.847623] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11869.886425] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11869.910502] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11875.962213] loop2: detected capacity change from 0 to 204800000 [11882.509839] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11882.721940] LustreError: fs22097-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11882.738784] LustreError: fs22097-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11882.753778] LustreError: 309153:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11882.763341] LustreError: 309153:0:(tgt_mount.c:2128:server_put_super()) no obd fs22097-OST0001 [11882.782307] LustreError: 309153:0:(tgt_mount.c:116:server_deregister_mount()) fs22097-OST0001 not registered [11882.838936] Lustre: server umount fs22097-OST0001 complete [11882.843185] LustreError: 309153:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11895.208244] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11895.455566] Lustre: fs22097-OST0001: new disk, initializing [11895.459496] Lustre: srv-fs22097-OST0001: No data found on store. Initialize space. [11895.470060] Lustre: 309705:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11895.585577] Lustre: fs22097-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11900.342282] LustreError: 304879:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11900.359278] LustreError: 304879:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [11902.801718] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11912.193305] Lustre: server umount fs22097-OST0000 complete [11922.318785] Lustre: server umount fs22097-OST0001 complete [11936.491438] loop1: detected capacity change from 0 to 204800000 [11943.216477] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11943.621134] Lustre: fs22097-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11950.952395] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11958.667105] loop2: detected capacity change from 0 to 204800000 [11965.390631] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11973.758995] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [11985.785909] Lustre: server umount fs22097-OST0000 complete [11996.812535] Lustre: server umount fs22097-OST0001 complete [11999.213950] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12004.335487] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12014.563989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12014.565927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12014.589271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12014.594583] Lustre: Skipped 1 previous similar message [12014.625017] Lustre: Skipped 1 previous similar message [12019.694403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12019.715187] Lustre: Skipped 1 previous similar message [12024.806558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12024.820444] Lustre: Skipped 1 previous similar message [12025.311198] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12025.561938] Lustre: server umount lustre-OST0000 complete [12031.969086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12031.974450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12031.998488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12035.048227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12036.624055] Lustre: server umount lustre-MDT0000 complete [12040.164396] LustreError: 302374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12040.198445] LustreError: 302374:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 72 previous similar messages [12042.183745] LustreError: 309704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862712 with bad export cookie 13021868098620000642 [12042.193755] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12042.199944] LustreError: 309704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12042.732567] Lustre: server umount lustre-MDT0001 complete [12056.009386] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [12059.480486] Key type lgssc unregistered [12059.842653] LNet: 315632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12059.848688] LNetError: 315632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12059.864930] LNet: Removed LNI 192.168.202.141@tcp [12060.728155] Key type .llcrypt unregistered [12060.731059] Key type ._llcrypt unregistered [12098.184089] Key type ._llcrypt registered [12098.186539] Key type .llcrypt registered [12098.291652] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [12116.777191] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [12118.255109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12118.448632] alg: No test for adler32 (adler32-zlib) [12119.632406] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [12120.000159] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [12121.799288] Key type lgssc registered [12123.238293] Lustre: Echo OBD driver; http://www.lustre.org/ [12163.789451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12163.804494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12165.075247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12165.126082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12165.240678] Lustre: lustre-MDT0000: new disk, initializing [12165.331076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12165.354552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12171.763468] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12186.559397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12186.832192] Lustre: 319491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12186.976816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12186.984375] Lustre: Skipped 1 previous similar message [12187.153502] Lustre: lustre-MDT0001: new disk, initializing [12187.308811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12187.382117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12187.405417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12193.579555] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12204.886933] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12212.710507] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12220.882645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12221.327407] Lustre: lustre-OST0000: new disk, initializing [12221.332901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12221.344954] Lustre: 321458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12221.504119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12227.706383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12227.728624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12227.908707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12228.111622] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12240.150740] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12242.196390] Lustre: Mounted lustre-client - version 2.17.58_39_ga87ca5d [12243.310873] Lustre: Unmounted lustre-client [12253.669927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12253.670751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12253.686991] LustreError: Skipped 1 previous similar message [12253.702814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12253.718050] Lustre: Skipped 1 previous similar message [12257.767639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12257.779548] Lustre: Skipped 1 previous similar message [12259.927567] Lustre: server umount lustre-OST0000 complete [12264.417036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12264.434962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12264.456840] Lustre: Skipped 1 previous similar message [12264.465738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12268.001701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12268.005131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12270.858264] Lustre: server umount lustre-MDT0000 complete [12274.911144] LustreError: 319483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788862945 with bad export cookie 9927975706032601759 [12274.924041] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12274.926886] LustreError: 319483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12275.356487] Lustre: server umount lustre-MDT0001 complete [12288.349884] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [12291.565874] Key type lgssc unregistered [12301.343231] LNet: 1 local NIs in recovery (showing 1): 192.168.202.141@tcp [12302.048316] LNet: 323619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12302.073331] LNetError: 323619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12371.760424] LNet: Removed LNI 192.168.202.141@tcp [12373.272912] Key type .llcrypt unregistered [12373.276599] Key type ._llcrypt unregistered [12394.444278] Key type ._llcrypt registered [12394.447627] Key type .llcrypt registered [12394.617785] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [12415.152142] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [12417.709111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12417.771064] alg: No test for adler32 (adler32-zlib) [12418.898139] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [12419.189937] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [12420.959284] Key type lgssc registered [12422.373390] Lustre: Echo OBD driver; http://www.lustre.org/ [12456.856426] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 06:25:26 (1788863126) [12463.355049] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12463.365106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12464.678938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12464.702729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12464.757788] Lustre: lustre-MDT0000: new disk, initializing [12464.871138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12464.900512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12470.922300] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12483.825392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12484.018820] Lustre: 326841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12484.092799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12484.105621] Lustre: Skipped 1 previous similar message [12484.248658] Lustre: lustre-MDT0001: new disk, initializing [12484.459742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12484.550049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12484.571183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12490.607074] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12501.918538] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12508.383393] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12516.576292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12516.944615] Lustre: lustre-OST0000: new disk, initializing [12516.952135] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12516.963265] Lustre: 328811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12517.090483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12523.085528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12523.095538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12523.222378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12524.393943] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12533.493767] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12554.211100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12554.217458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12554.217696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12554.227903] Lustre: Skipped 1 previous similar message [12554.254282] LustreError: Skipped 1 previous similar message [12557.482540] Lustre: server umount lustre-OST0000 complete [12564.456631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12564.461487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12564.477114] Lustre: Skipped 1 previous similar message [12564.491772] Lustre: Skipped 2 previous similar messages [12568.618604] Lustre: server umount lustre-MDT0000 complete [12572.553391] LustreError: 326836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788863243 with bad export cookie 9428170898205215864 [12572.561755] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12572.566102] LustreError: 326836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12573.139119] Lustre: server umount lustre-MDT0001 complete [12585.554640] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [12588.680542] Key type lgssc unregistered [12589.067201] LNet: 330719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12589.076679] LNetError: 330719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12589.104245] LNet: Removed LNI 192.168.202.141@tcp [12589.971212] Key type .llcrypt unregistered [12589.975963] Key type ._llcrypt unregistered [12613.331156] Key type ._llcrypt registered [12613.332909] Key type .llcrypt registered [12613.426474] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [12615.169568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12615.232592] alg: No test for adler32 (adler32-zlib) [12616.424845] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [12616.686892] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [12618.344678] Key type lgssc registered [12619.968909] Lustre: Echo OBD driver; http://www.lustre.org/ [12628.196168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12628.227099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12629.803852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12634.646570] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12640.228593] LustreError: 331990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12640.241980] LustreError: 331990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12643.649159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12644.082341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12648.998881] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12657.693634] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12665.514928] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12673.094131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12673.522303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12680.336942] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12690.049642] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12701.860390] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12703.297182] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12708.317348] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [12709.346389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12709.351513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12709.362313] Lustre: Skipped 2 previous similar messages [12709.374332] Lustre: Skipped 2 previous similar messages [12713.401412] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.41@tcp (stopping) [12714.050622] Lustre: server umount lustre-MDT0000 complete [12714.465728] LustreError: 331989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12714.489718] LustreError: 331989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12718.513682] LustreError: 331984:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12718.515680] LustreError: 331970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788863389 with bad export cookie 11922011545178379981 [12718.545299] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12718.571198] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.41@tcp (stopping) [12719.585815] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12723.618626] LustreError: 333829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12723.619049] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12723.640341] LustreError: 333829:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12723.655991] Lustre: Skipped 2 previous similar messages [12725.068872] Lustre: server umount lustre-MDT0001 complete [12729.054814] Lustre: server umount lustre-OST0000 complete [12738.141481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12743.384215] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12746.590987] Lustre: 336170:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.141@tcp [12748.745542] Lustre: server umount MGS complete [12757.502950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12757.982213] LustreError: 336743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12758.044494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12762.216682] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12768.233640] LustreError: 336743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12768.259855] LustreError: 336743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12772.225596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12772.797884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12777.925160] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12788.145611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12788.427040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12795.441508] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12804.630936] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12806.457186] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12816.309831] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12819.007834] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12828.648559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12828.658310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12828.663220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12828.677474] LustreError: Skipped 1 previous similar message [12828.684587] Lustre: Skipped 1 previous similar message [12832.050167] Lustre: server umount lustre-OST0000 complete [12838.880979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12838.899024] Lustre: Skipped 2 previous similar messages [12849.121138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12849.123858] Lustre: Skipped 4 previous similar messages [12850.143154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12850.558301] Lustre: server umount lustre-MDT0000 complete [12854.247719] LustreError: 336743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12854.290455] LustreError: 336743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12855.601755] LustreError: 336724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788863526 with bad export cookie 11922011545178380821 [12855.608221] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12855.623495] LustreError: 336724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12856.157911] Lustre: server umount lustre-MDT0001 complete [12869.747176] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [12872.969881] Key type lgssc unregistered [12873.287627] LNet: 340185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12873.291365] LNetError: 340185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12873.319868] LNet: Removed LNI 192.168.202.141@tcp [12874.123301] Key type .llcrypt unregistered [12874.125887] Key type ._llcrypt unregistered [12893.209736] Key type ._llcrypt registered [12893.212496] Key type .llcrypt registered [12893.325341] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [12911.199302] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [12912.563604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12912.807327] alg: No test for adler32 (adler32-zlib) [12913.987870] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [12914.347107] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [12916.088087] Key type lgssc registered [12917.503962] Lustre: Echo OBD driver; http://www.lustre.org/ [12965.025588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12965.058293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12966.481616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12966.537623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12966.607734] Lustre: lustre-MDT0000: new disk, initializing [12966.732520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12966.749040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12971.957869] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [12984.638866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12984.828495] Lustre: 343343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12984.962737] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12984.971402] Lustre: Skipped 1 previous similar message [12985.121602] Lustre: lustre-MDT0001: new disk, initializing [12985.279719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12985.318957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12985.344480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12990.314691] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13000.762639] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13008.547160] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13015.505387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13015.913182] Lustre: lustre-OST0000: new disk, initializing [13015.922279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13015.932363] Lustre: 345312:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13016.055977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13022.771344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13022.779670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13022.897757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13024.255733] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13035.614220] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13042.961752] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13043.292645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13047.928533] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13048.122832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13053.930184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13053.936940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13053.957344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13053.969313] Lustre: Skipped 1 previous similar message [13056.692523] Lustre: server umount lustre-OST0000 complete [13062.112291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13062.121135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13062.143522] Lustre: Skipped 1 previous similar message [13062.146612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13064.162475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13064.165879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13064.183979] Lustre: Skipped 1 previous similar message [13067.093860] Lustre: server umount lustre-MDT0000 complete [13071.096275] LustreError: 343335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788863741 with bad export cookie 3976060537213968688 [13071.102206] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13071.114781] LustreError: 343335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13071.476234] Lustre: server umount lustre-MDT0001 complete [13082.551784] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 06:35:52 (1788863752) [13106.761264] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [13121.775713] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [13169.574163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13169.878291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13169.904389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13170.034952] Lustre: lustre-MDT0000: new disk, initializing [13170.152968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13170.185857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13175.563491] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13188.753446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13188.902611] Lustre: 350098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13188.977834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13188.984673] Lustre: Skipped 1 previous similar message [13189.058424] Lustre: lustre-MDT0001: new disk, initializing [13189.194348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13189.261554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13189.281581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13195.434278] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13206.368560] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13212.440388] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13220.316420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13220.722691] Lustre: lustre-OST0000: new disk, initializing [13220.730528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13220.740524] Lustre: 352066:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13220.935837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13228.777978] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [13230.677244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13230.695289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13230.839668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13240.876213] Lustre: DEBUG MARKER: oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13282.274366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13282.283629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13282.299985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13282.328342] Lustre: Skipped 1 previous similar message [13282.361692] Lustre: Skipped 1 previous similar message [13287.124864] Lustre: server umount lustre-OST0000 complete [13292.517112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13292.521353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13292.550184] Lustre: Skipped 1 previous similar message [13292.568207] Lustre: Skipped 1 previous similar message [13298.032147] Lustre: server umount lustre-MDT0000 complete [13302.752828] LustreError: 350110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13302.791421] LustreError: 350110:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13303.091953] LustreError: 350091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788863973 with bad export cookie 3976060537213969570 [13303.101574] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13303.105054] LustreError: 350091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13303.703759] Lustre: server umount lustre-MDT0001 complete [13318.946488] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13322.258579] Key type lgssc unregistered [13322.595044] LNet: 354239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13322.613403] LNetError: 354239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13322.637319] LNet: Removed LNI 192.168.202.141@tcp [13323.701186] Key type .llcrypt unregistered [13323.703434] Key type ._llcrypt unregistered [13432.065309] Key type ._llcrypt registered [13432.069580] Key type .llcrypt registered [13432.185328] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13432.690458] Key type .llcrypt unregistered [13432.697067] Key type ._llcrypt unregistered [13527.047208] Key type ._llcrypt registered [13527.048906] Key type .llcrypt registered [13527.274155] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13527.690379] Key type .llcrypt unregistered [13527.693261] Key type ._llcrypt unregistered [13568.010587] Key type ._llcrypt registered [13568.021818] Key type .llcrypt registered [13568.177792] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13568.639432] Key type .llcrypt unregistered [13568.641829] Key type ._llcrypt unregistered [13605.613340] Key type ._llcrypt registered [13605.621442] Key type .llcrypt registered [13605.816960] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13606.371510] Key type .llcrypt unregistered [13606.373190] Key type ._llcrypt unregistered [13695.600942] Key type ._llcrypt registered [13695.604732] Key type .llcrypt registered [13695.743654] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13696.187321] Key type .llcrypt unregistered [13696.188792] Key type ._llcrypt unregistered [13736.577961] Key type ._llcrypt registered [13736.586933] Key type .llcrypt registered [13736.758257] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13737.217960] Key type .llcrypt unregistered [13737.227869] Key type ._llcrypt unregistered [13774.178339] Key type ._llcrypt registered [13774.186536] Key type .llcrypt registered [13774.414504] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13774.900272] Key type .llcrypt unregistered [13774.904722] Key type ._llcrypt unregistered [13813.541903] Key type ._llcrypt registered [13813.550261] Key type .llcrypt registered [13813.702382] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13814.028669] Key type .llcrypt unregistered [13814.031148] Key type ._llcrypt unregistered [13846.416138] Key type ._llcrypt registered [13846.419500] Key type .llcrypt registered [13846.533577] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13846.902588] Key type .llcrypt unregistered [13846.911365] Key type ._llcrypt unregistered [13880.666312] Key type ._llcrypt registered [13880.668106] Key type .llcrypt registered [13880.809657] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13881.195090] Key type .llcrypt unregistered [13881.202053] Key type ._llcrypt unregistered [13965.383414] Key type ._llcrypt registered [13965.397083] Key type .llcrypt registered [13965.555271] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [13966.058376] Key type .llcrypt unregistered [13966.063914] Key type ._llcrypt unregistered [14005.920927] Key type ._llcrypt registered [14005.922246] Key type .llcrypt registered [14006.038322] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [14006.416739] Key type .llcrypt unregistered [14006.418265] Key type ._llcrypt unregistered [14045.648514] Key type ._llcrypt registered [14045.652631] Key type .llcrypt registered [14045.797461] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [14046.219535] Key type .llcrypt unregistered [14046.222987] Key type ._llcrypt unregistered [14108.212989] Key type ._llcrypt registered [14108.217790] Key type .llcrypt registered [14108.449774] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [14128.789915] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [14130.378195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14130.647840] alg: No test for adler32 (adler32-zlib) [14132.034895] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [14132.486285] LNet: Added LNI 192.168.202.141@tcp [8/256/0/180] [14134.215362] Key type lgssc registered [14135.758178] Lustre: Echo OBD driver; http://www.lustre.org/ [14178.196095] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14178.235954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14179.622678] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14179.677508] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14179.777555] Lustre: tst250fs-MDT0000: new disk, initializing [14179.907917] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14179.947774] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14185.859075] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [14200.093162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14200.195398] Lustre: 374572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14200.246498] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14200.250261] Lustre: Skipped 1 previous similar message [14200.308227] Lustre: tst250fs-MDT0001: new disk, initializing [14200.481210] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14200.532928] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14200.544933] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14206.765085] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [14212.126128] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14221.166430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14221.583110] Lustre: tst250fs-OST0000: new disk, initializing [14221.594983] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14221.608832] Lustre: 376206:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14221.812609] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14229.068073] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14229.078609] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14229.197370] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [14229.213167] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14240.604991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14240.859646] Lustre: tst250fs-OST0001: new disk, initializing [14240.867349] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14240.878791] Lustre: 377076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14240.998570] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14247.177410] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_default_debug -1 all [14250.619553] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14250.638176] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14250.820637] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14253.546953] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14255.758312] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14255.766780] Lustre: Skipped 1 previous similar message [14281.698720] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14281.710609] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14281.745869] Lustre: Skipped 2 previous similar messages [14281.769144] Lustre: Skipped 3 previous similar messages [14285.730902] Lustre: server umount tst250fs-MDT0000 complete [14290.520796] LustreError: 374563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788864961 with bad export cookie 13390841061178756212 [14290.522340] LustreError: MGC192.168.202.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14290.530270] LustreError: 374563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14291.021414] Lustre: server umount tst250fs-MDT0001 complete [14306.424658] Lustre: server umount tst250fs-OST0000 complete [14308.198685] Lustre: 372793:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788864962/real 1788864962] req@ffff8eda3fb35500 x1875760699740800/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788864978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14308.220914] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14308.230164] Lustre: Skipped 1 previous similar message [14311.372140] Lustre: server umount tst250fs-OST0001 complete [14334.738164] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [14346.219208] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [14386.254613] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 06:57:35 (1788865055) [14388.356074] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14409.986765] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing set_hostid [14421.778782] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing load_modules_local [14456.079522] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14131 sec ========= 06:58:46 (1788865126) [14458.391217] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:58:47 (1788865127) === [14463.362410] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:58:52 (1788865132) === [14511.593841] Lustre: DEBUG MARKER: oleg241-server.virtnet: executing unload_modules_local [14515.020304] Key type lgssc unregistered [14515.340686] LNet: 386223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14515.351063] LNetError: 386223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14516.393722] LNet: Removed LNI 192.168.202.141@tcp [14517.677257] Key type .llcrypt unregistered [14517.683137] Key type ._llcrypt unregistered