[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493405935 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a2000-0x1465ccfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003164] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008036] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.011030] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013043] SELinux: Initializing. [ 0.015038] *** VALIDATE selinux *** [ 0.023708] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028714] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031146] *** VALIDATE tmpfs *** [ 0.033058] *** VALIDATE proc *** [ 0.034241] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036254] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.042824] debug: unmapping init [mem 0xffffffffb5c59000-0xffffffffb5c60fff] [ 0.044212] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045786] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052017] ... event mask: 000000070000000f [ 0.053344] rcu: Hierarchical SRCU implementation. [ 0.055533] smp: Bringing up secondary CPUs ... [ 0.056614] x86: Booting SMP configuration: [ 0.057046] .... node #0, CPUs: #1 #2 #3 [ 0.060487] smp: Brought up 1 node, 4 CPUs [ 0.062026] smpboot: Max logical packages: 1 [ 0.063022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145030] node 0 deferred pages initialised in 79ms [ 0.149124] devtmpfs: initialized [ 0.150249] x86/mm: Memory block size: 128MB [ 0.153708] gcov: version magic: 0x41383552 [ 0.155292] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157259] pinctrl core: initialized pinctrl subsystem [ 0.158178] [ 0.158589] ************************************************************* [ 0.159016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160014] ** ** [ 0.161015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162014] ** ** [ 0.163014] ** This means that this kernel is built to expose internal ** [ 0.164014] ** IOMMU data structures, which may compromise security on ** [ 0.165013] ** your system. ** [ 0.166012] ** ** [ 0.167014] ** If you see this message and you are not debugging the ** [ 0.168013] ** kernel, report this immediately to your vendor! ** [ 0.169013] ** ** [ 0.170015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171013] ************************************************************* [ 0.172756] NET: Registered protocol family 16 [ 0.173000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.179071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.183012] cpuidle: using governor menu [ 0.185123] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186499] PCI: Using configuration type 1 for base access [ 0.187145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195110] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197072] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.201080] cryptd: max_cpu_qlen set to 1000 [ 0.204770] ACPI: Added _OSI(Module Device) [ 0.207015] ACPI: Added _OSI(Processor Device) [ 0.208015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.210016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.215230] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.222310] ACPI: Interpreter enabled [ 0.224060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.225012] ACPI: Using IOAPIC for interrupt routing [ 0.227136] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231385] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.241974] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.244050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.246021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.249093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254453] acpiphp: Slot [2] registered [ 0.256142] acpiphp: Slot [5] registered [ 0.257116] acpiphp: Slot [6] registered [ 0.259121] acpiphp: Slot [7] registered [ 0.260111] acpiphp: Slot [8] registered [ 0.261137] acpiphp: Slot [9] registered [ 0.263146] acpiphp: Slot [10] registered [ 0.265133] acpiphp: Slot [3] registered [ 0.267104] acpiphp: Slot [4] registered [ 0.268095] acpiphp: Slot [11] registered [ 0.269108] acpiphp: Slot [12] registered [ 0.271119] acpiphp: Slot [13] registered [ 0.272146] acpiphp: Slot [14] registered [ 0.273112] acpiphp: Slot [15] registered [ 0.275113] acpiphp: Slot [16] registered [ 0.276104] acpiphp: Slot [17] registered [ 0.278121] acpiphp: Slot [18] registered [ 0.279105] acpiphp: Slot [19] registered [ 0.280121] acpiphp: Slot [20] registered [ 0.282101] acpiphp: Slot [21] registered [ 0.283110] acpiphp: Slot [22] registered [ 0.285112] acpiphp: Slot [23] registered [ 0.287118] acpiphp: Slot [24] registered [ 0.288125] acpiphp: Slot [25] registered [ 0.290123] acpiphp: Slot [26] registered [ 0.291122] acpiphp: Slot [27] registered [ 0.293113] acpiphp: Slot [28] registered [ 0.294117] acpiphp: Slot [29] registered [ 0.296106] acpiphp: Slot [30] registered [ 0.297107] acpiphp: Slot [31] registered [ 0.299107] PCI host bridge to bus 0000:00 [ 0.300020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.303026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.305025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.308028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.310033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.313027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.315175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.318074] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.322000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.331682] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.336370] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.338019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.341023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.343019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345559] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.348798] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.351041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.354841] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.360017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.375023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378887] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.384821] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.396021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.402020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.434886] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.451020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.488024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.500009] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.512031] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.520041] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.542027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.557813] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.567018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.573018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.591019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.601301] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608031] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.617025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.637033] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.647297] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.656020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.669023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.696018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.712401] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.714483] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.717787] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.720545] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.723360] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.729213] iommu: Default domain type: Passthrough [ 0.732256] SCSI subsystem initialized [ 0.734242] ACPI: bus type USB registered [ 0.736104] usbcore: registered new interface driver usbfs [ 0.738084] usbcore: registered new interface driver hub [ 0.740118] usbcore: registered new device driver usb [ 0.742207] pps_core: LinuxPPS API ver. 1 registered [ 0.744014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.747089] PTP clock support registered [ 0.749321] EDAC MC: Ver: 3.0.0 [ 0.751130] PCI: Using ACPI for IRQ routing [ 0.753950] NetLabel: Initializing [ 0.755012] NetLabel: domain hash size = 128 [ 0.756011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.758090] NetLabel: unlabeled traffic allowed by default [ 0.760159] vgaarb: loaded [ 0.761293] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.763016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.768343] clocksource: Switched to clocksource kvm-clock [ 0.878255] VFS: Disk quotas dquot_6.6.0 [ 0.880114] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.883131] *** VALIDATE ramfs *** [ 0.884753] *** VALIDATE hugetlbfs *** [ 0.887235] pnp: PnP ACPI init [ 0.889629] pnp: PnP ACPI: found 6 devices [ 0.910419] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.914048] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.916160] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.917899] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.920609] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.922896] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.925916] NET: Registered protocol family 2 [ 0.928434] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.933144] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.936457] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.940948] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.945565] TCP: Hash tables configured (established 65536 bind 65536) [ 0.948934] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.952312] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.955041] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.957861] NET: Registered protocol family 1 [ 0.960436] RPC: Registered named UNIX socket transport module. [ 0.962742] RPC: Registered udp transport module. [ 0.964657] RPC: Registered tcp transport module. [ 0.966594] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.969321] NET: Registered protocol family 44 [ 0.971075] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.973532] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.975729] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.978270] PCI: CLS 0 bytes, default 64 [ 0.980056] Unpacking initramfs... [ 2.382629] debug: unmapping init [mem 0xffff94ffbcc54000-0xffff94ffbffbffff] [ 2.386811] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.389259] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.391922] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.927863] Initialise system trusted keyrings [ 2.929777] Key type blacklist registered [ 2.931899] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.942103] zbud: loaded [ 2.945440] *** VALIDATE nfs *** [ 2.946877] *** VALIDATE nfs4 *** [ 2.948883] pstore: using deflate compression [ 2.953274] Platform Keyring initialized [ 3.053460] NET: Registered protocol family 38 [ 3.055561] Key type asymmetric registered [ 3.057187] Asymmetric key parser 'x509' registered [ 3.059241] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.062548] io scheduler mq-deadline registered [ 3.064528] io scheduler kyber registered [ 3.066301] io scheduler bfq registered [ 3.068235] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.071431] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.074442] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.077463] ACPI: Power Button [PWRF] [ 3.082671] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.089555] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.112501] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.121353] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.137556] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.165781] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.195529] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.200850] Non-volatile memory driver v1.3 [ 3.202814] Linux agpgart interface v0.103 [ 3.240343] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.243573] vda: detected capacity change from 0 to 75100160 [ 3.262814] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.265918] vdb: detected capacity change from 0 to 1073741824 [ 3.286370] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.289797] vdc: detected capacity change from 0 to 2621440000 [ 3.307526] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.310628] vdd: detected capacity change from 0 to 2621440000 [ 3.328529] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.331593] vde: detected capacity change from 0 to 4294967296 [ 3.347112] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.349877] vdf: detected capacity change from 0 to 4294967296 [ 3.359550] libphy: Fixed MDIO Bus: probed [ 3.369711] usbcore: registered new interface driver usbserial_generic [ 3.372612] usbserial: USB Serial support registered for generic [ 3.374952] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.379519] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.381247] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.383807] mousedev: PS/2 mouse device common for all mice [ 3.387188] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.388652] rtc_cmos 00:05: RTC can wake from S4 [ 3.394668] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.394716] rtc_cmos 00:05: registered as rtc0 [ 3.399736] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.402830] intel_pstate: CPU model not supported [ 3.405449] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.408205] hid: raw HID events driver (C) Jiri Kosina [ 3.411171] usbcore: registered new interface driver usbhid [ 3.413012] usbhid: USB HID core driver [ 3.414566] drop_monitor: Initializing network drop monitor service [ 3.416934] Initializing XFRM netlink socket [ 3.418929] NET: Registered protocol family 10 [ 3.421788] Segment Routing with IPv6 [ 3.423254] NET: Registered protocol family 17 [ 3.425743] mpls_gso: MPLS GSO support [ 3.432963] RAS: Correctable Errors collector initialized. [ 3.435303] AVX version of gcm_enc/dec engaged. [ 3.437216] AES CTR mode by8 optimization enabled [ 3.514917] sched_clock: Marking stable (3514875094, 0)->(4453981487, -939106393) [ 3.518857] registered taskstats version 1 [ 3.521323] Loading compiled-in X.509 certificates [ 3.523398] zswap: loaded using pool lzo/zbud [ 3.547523] Key type big_key registered [ 3.561504] Key type encrypted registered [ 3.563166] ima: No TPM chip found, activating TPM-bypass! [ 3.565130] ima: Allocated hash algorithm: sha1 [ 3.567284] ima: No architecture policies found [ 3.569134] evm: Initialising EVM extended attributes: [ 3.571067] evm: security.selinux [ 3.572315] evm: security.ima [ 3.573514] evm: security.capability [ 3.574828] evm: HMAC attrs: 0x1 [ 3.577527] rtc_cmos 00:05: setting system clock to 2026-09-08 02:07:03 UTC (1788833223) [ 3.583872] debug: unmapping init [mem 0xffffffffb6c03000-0xffffffffb6dfffff] [ 3.587408] debug: unmapping init [mem 0xffffffffb5982000-0xffffffffb5c58fff] [ 3.596127] Write protecting the kernel read-only data: 28672k [ 3.599586] debug: unmapping init [mem 0xffffffffb4003000-0xffffffffb41fffff] [ 3.602428] debug: unmapping init [mem 0xffffffffb4914000-0xffffffffb49fffff] [ 3.636574] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.646915] systemd[1]: Detected virtualization kvm. [ 3.648882] systemd[1]: Detected architecture x86-64. [ 3.650923] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.679374] systemd[1]: No hostname configured. [ 3.681688] systemd[1]: Set hostname to . [ 3.684326] random: systemd: uninitialized urandom read (16 bytes read) [ 3.687227] systemd[1]: Initializing machine ID from random generator. [ 3.741407] random: ln: uninitialized urandom read (6 bytes read) [ 3.829732] random: systemd: uninitialized urandom read (16 bytes read) [ 3.832613] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.837331] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.842891] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.514704] device-mapper: uevent: version 1.0.3 [ 4.517463] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [[ 4.778985] random: fast init done  OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.277232] virtio_net virtio0 ens2: renamed from eth0 [ 5.357237] scsi host0: ata_piix [ 5.367206] scsi host1: ata_piix [ 5.382885] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.385766] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.897206] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.080876] random: crng init done [ 10.082524] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.579803] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.817394] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.080365] SELinux: Disabled at runtime. [ 12.143571] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.152651] systemd[1]: Detected virtualization kvm. [ 12.153758] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.651981] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.655729] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.661700] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.666509] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.671205] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.679416] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.684145] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ 12.794568] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Huge Pages File System... Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.084273] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.461967] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.497144] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.630534] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.642287] EDAC sbridge: Ver: 1.1.2 [ 15.384480] Key type dns_resolver registered [ 15.698240] NFS: Registering the id_resolver key type [ 15.700050] Key type id_resolver registered [ 15.701698] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg143-server login: [ 69.176742] libcfs: loading out-of-tree module taints kernel. [ 69.267438] Key type ._llcrypt registered [ 69.270958] Key type .llcrypt registered [ 69.525418] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 90.543227] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 92.559559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 92.575568] alg: No test for adler32 (adler32-zlib) [ 94.203757] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 95.354804] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [ 97.144183] Key type lgssc registered [ 99.140911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 122.512310] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.605305] hrtimer: interrupt took 6347388 ns [ 177.635294] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 193.504639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 193.569045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.856604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 194.906829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 195.054609] Lustre: lustre-MDT0000: new disk, initializing [ 195.243048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.271100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 200.086446] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.061129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.215099] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.249676] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.255820] Lustre: Skipped 1 previous similar message [ 216.381754] Lustre: lustre-MDT0001: new disk, initializing [ 216.489301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.557079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 216.593501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 221.580979] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.749573] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 240.260169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.675050] Lustre: lustre-OST0000: new disk, initializing [ 240.678431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 240.680946] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.806104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 249.260606] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.962666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 249.978055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 250.208049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 270.884321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.127530] Lustre: lustre-OST0001: new disk, initializing [ 271.139514] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 271.145991] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.262906] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 278.914209] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 280.751503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 280.760471] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 280.887569] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 293.762528] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 303.776576] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 312.226308] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing check_logdir /tmp/testlogs/ [ 318.603622] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing yml_node [ 323.368655] Lustre: DEBUG MARKER: Client: 2.17.57.106 [ 326.764037] Lustre: DEBUG MARKER: MDS: 2.17.57.106 [ 329.951358] Lustre: DEBUG MARKER: OSS: 2.17.57.106 [ 332.127410] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 22:12:28 EDT 2026 [ 354.065503] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 356.063100] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 375.265781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 375.271849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.314127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 378.348775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 378.348915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.359901] Lustre: Skipped 1 previous similar message [ 378.390088] Lustre: Skipped 1 previous similar message [ 379.884695] Lustre: server umount lustre-MDT0000 complete [ 383.949822] LustreError: 10341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833603 with bad export cookie 13575048109283543020 [ 383.954251] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 383.960182] LustreError: 10341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 388.577278] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 388.578315] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.578484] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 388.578489] Lustre: Skipped 2 previous similar messages [ 388.597381] Lustre: Skipped 2 previous similar messages [ 388.651966] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 393.701066] LustreError: 8159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 393.702536] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 393.738022] Lustre: Skipped 2 previous similar messages [ 393.741352] LustreError: 8159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 398.304637] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 398.793312] Lustre: server umount lustre-MDT0001 complete [ 404.100982] Lustre: server umount lustre-OST0000 complete [ 408.763839] Lustre: server umount lustre-OST0001 complete [ 417.131233] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 430.586511] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 477.931739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.225569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 478.258674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 478.376371] Lustre: lustre-MDT0000: new disk, initializing [ 478.528138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.549649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 483.804410] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 495.831957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.961884] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 496.020386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 496.023781] Lustre: Skipped 1 previous similar message [ 496.145463] Lustre: lustre-MDT0001: new disk, initializing [ 496.253123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 496.310337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 496.333518] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 500.050498] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 513.197721] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 520.980178] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 529.467736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 529.792837] Lustre: lustre-OST0000: new disk, initializing [ 529.797363] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 529.809943] Lustre: 17055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 529.978464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.679560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 531.706542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 531.843282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 536.491686] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 547.756027] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 552.660197] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 552.876105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 558.259865] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 558.499887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 562.669355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.671978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 562.686064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 566.554758] Lustre: server umount lustre-OST0000 complete [ 572.902296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.904690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 572.924871] Lustre: Skipped 1 previous similar message [ 572.935029] LustreError: Skipped 1 previous similar message [ 572.961217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 572.971454] Lustre: Skipped 1 previous similar message [ 577.126857] Lustre: server umount lustre-MDT0000 complete [ 578.024305] LustreError: 15097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.068760] LustreError: 15097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 582.027176] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833801 with bad export cookie 13575048109283544553 [ 582.032728] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 582.053423] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 582.531599] Lustre: server umount lustre-MDT0001 complete [ 592.238518] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:16:48 (1788833808) [ 619.406268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.157559] LustreError: 19931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 620.322130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.636543] LustreError: 19932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.943299] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 635.883241] LustreError: 19931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.901152] LustreError: 19931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 638.697426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.420889] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 654.192580] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 663.228756] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 667.556146] Lustre: Failing over lustre-MDT0000 [ 667.971950] Lustre: server umount lustre-MDT0000 complete [ 670.202727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.214519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 670.231950] Lustre: Skipped 1 previous similar message [ 670.240090] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.308515] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 686.570956] Lustre: 3642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788833890/real 1788833890] req@ffff94ff03381180 x1875727682284032/t0(0) o400->MGC192.168.201.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788833906 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 686.622668] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 692.846317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.806132] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9500350ded80 x1875727682286976/t0(0) o250->MGC192.168.201.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 697.173926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 697.184455] Lustre: Skipped 1 previous similar message [ 697.262851] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 702.448781] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 702.469392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 702.493594] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 703.452609] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 714.923783] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 722.921195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 722.937522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.963634] Lustre: Skipped 1 previous similar message [ 722.970853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 726.300853] Lustre: server umount lustre-MDT0000 complete [ 730.084935] LustreError: 19932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 730.117940] LustreError: 19932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 730.532473] LustreError: 19912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833950 with bad export cookie 13575048109283545967 [ 730.536771] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 730.547984] LustreError: 19912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 731.103170] Lustre: server umount lustre-MDT0001 complete [ 740.307529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 746.296555] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 756.498614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 762.531511] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 773.065413] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 780.858519] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 784.311258] Lustre: Failing over lustre-MDT0000 [ 784.611726] Lustre: server umount lustre-MDT0000 complete [ 786.936690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.969957] Lustre: Skipped 1 previous similar message [ 797.155267] LustreError: 23270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 797.183289] LustreError: 23270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 802.208150] Lustre: 3642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834006/real 1788834006] req@ffff95002e163800 x1875727682317056/t0(0) o400->MGC192.168.201.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788834022 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 802.254977] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 805.452203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.512870] LustreError: 3638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9500385d8000 x1875727682319872/t0(0) o250->MGC192.168.201.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 813.009131] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 818.148289] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 818.152970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 818.180868] Lustre: Skipped 1 previous similar message [ 818.191624] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 819.028381] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 832.882747] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 838.625967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 838.629952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.631990] Lustre: Skipped 2 previous similar messages [ 844.547700] Lustre: server umount lustre-MDT0000 complete [ 850.156810] LustreError: 24012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834070 with bad export cookie 13575048109283546905 [ 850.169107] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 850.171263] LustreError: 24012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 850.671679] Lustre: server umount lustre-MDT0001 complete [ 861.773250] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:21:18 (1788834078) [ 881.238866] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 894.000646] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 941.588114] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 954.330347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.561276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 954.601172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 954.682788] Lustre: lustre-MDT0000: new disk, initializing [ 954.763963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.768874] Lustre: Skipped 3 previous similar messages [ 954.786190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 960.213043] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 972.056655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 972.199127] Lustre: 29655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 972.591810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 978.034249] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 989.794180] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 997.489359] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1005.254019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1005.713866] Lustre: 31624:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1007.775088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1007.818591] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1007.879983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1007.908427] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1013.589444] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 1024.951917] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1031.644102] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1059.311583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.328749] Lustre: Skipped 3 previous similar messages [ 1059.333483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1059.333823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1059.342482] Lustre: Skipped 3 previous similar messages [ 1061.020256] Lustre: server umount lustre-OST0000 complete [ 1071.110961] Lustre: server umount lustre-MDT0000 complete [ 1074.657586] LustreError: 29666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1074.684391] LustreError: 29666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 1075.155154] LustreError: 29647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834295 with bad export cookie 13575048109283547311 [ 1075.163323] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1075.173438] LustreError: 29647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1075.662752] Lustre: server umount lustre-MDT0001 complete [ 1088.391901] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [ 1092.614476] Key type lgssc unregistered [ 1093.005387] LNet: 33604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1093.037021] LNetError: 33604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1094.056078] LNet: Removed LNI 192.168.201.143@tcp [ 1095.649345] Key type .llcrypt unregistered [ 1095.652571] Key type ._llcrypt unregistered [ 1127.519975] Key type ._llcrypt registered [ 1127.524499] Key type .llcrypt registered [ 1127.691467] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 1129.237036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1129.309143] alg: No test for adler32 (adler32-zlib) [ 1130.359917] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 1130.686416] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [ 1132.392241] Key type lgssc registered [ 1133.591823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1142.046733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1142.060487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.577321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.947749] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 1154.029615] LustreError: 35016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1154.048537] LustreError: 35016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1159.142125] LustreError: 35017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1159.794135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.056397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1164.274655] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 1173.425187] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1180.817212] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1189.935480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1190.428514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.304232] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 1204.805032] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1209.231870] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1214.027639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1233.492791] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:27:29 (1788834449) [ 1236.115147] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1238.692519] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1247.122459] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:27:43 (1788834463) [ 1257.529940] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:27:54 (1788834474) [ 1258.750504] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1267.639921] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1267.647584] Lustre: Skipped 2 previous similar messages [ 1269.986227] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:28:06 (1788834486) [ 1271.844642] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1271.848961] Lustre: Skipped 2 previous similar messages [ 1279.989458] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1279.994873] Lustre: Skipped 4 previous similar messages [ 1296.809985] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:28:33 (1788834513) [ 1300.407307] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1304.586962] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1314.979234] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:28:51 (1788834531) [ 1316.808923] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1316.823328] Lustre: Skipped 6 previous similar messages [ 1331.142416] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:29:07 (1788834547) [ 1352.166808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.167985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1352.202680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1356.862558] Lustre: server umount lustre-OST0000 complete [ 1362.407416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.419387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1362.423665] Lustre: Skipped 2 previous similar messages [ 1362.441592] Lustre: Skipped 2 previous similar messages [ 1367.522991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1367.532370] Lustre: Skipped 1 previous similar message [ 1372.642222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1372.653501] Lustre: Skipped 1 previous similar message [ 1375.200152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1375.340513] Lustre: server umount lustre-MDT0000 complete [ 1377.762693] LustreError: 35011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1377.787154] LustreError: 35011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1379.031654] LustreError: 36861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834598 with bad export cookie 17866203368342642870 [ 1379.037345] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1379.041788] LustreError: 36861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1379.421653] Lustre: server umount lustre-MDT0001 complete [ 1391.879411] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [ 1394.589426] Key type lgssc unregistered [ 1394.858550] LNet: 42903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.865889] LNetError: 42903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.894167] LNet: Removed LNI 192.168.201.143@tcp [ 1395.840380] Key type .llcrypt unregistered [ 1395.844407] Key type ._llcrypt unregistered [ 1415.159581] Key type ._llcrypt registered [ 1415.161744] Key type .llcrypt registered [ 1415.256851] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 1431.264338] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 1432.789079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1432.900146] alg: No test for adler32 (adler32-zlib) [ 1433.890379] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 1434.106068] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [ 1435.784329] Key type lgssc registered [ 1436.785661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1472.425649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1472.460680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.804791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1473.864765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1474.070208] Lustre: lustre-MDT0000: new disk, initializing [ 1474.231344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1474.252530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1480.064261] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 1491.367533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1491.520384] Lustre: 46057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1491.606359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1491.610550] Lustre: Skipped 1 previous similar message [ 1491.761650] Lustre: lustre-MDT0001: new disk, initializing [ 1491.949395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1492.000231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1492.015950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1497.298457] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 1508.600160] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1515.195308] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1523.759796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1524.240262] Lustre: lustre-OST0000: new disk, initializing [ 1524.245329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1524.255892] Lustre: 48024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1524.382130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1532.755123] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 1533.481706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1533.495448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1533.601182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1543.636053] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1548.329975] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1548.610675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1554.333593] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1554.511568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1559.524601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1559.526054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1559.570280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1559.585615] Lustre: Skipped 1 previous similar message [ 1564.647944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1564.656200] Lustre: Skipped 1 previous similar message [ 1569.772025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.781018] Lustre: Skipped 1 previous similar message [ 1571.296130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1571.611866] Lustre: server umount lustre-OST0000 complete [ 1578.977455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1578.988938] LustreError: Skipped 1 previous similar message [ 1578.994230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.007882] Lustre: Skipped 1 previous similar message [ 1579.017604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1580.023204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1585.126537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1585.136972] Lustre: Skipped 3 previous similar messages [ 1590.240193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1590.443271] Lustre: server umount lustre-MDT0000 complete [ 1594.964658] LustreError: 46047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834814 with bad export cookie 4546249071056472130 [ 1594.978194] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1594.993388] LustreError: 46047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1595.310661] Lustre: server umount lustre-MDT0001 complete [ 1607.564985] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:33:44 (1788834824) [ 1614.916570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.328045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.325452] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 1620.453322] LustreError: 50488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1620.484602] LustreError: 50488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1625.571899] LustreError: 50489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1630.526094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.690205] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1630.693301] Lustre: Skipped 2 previous similar messages [ 1630.893572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1635.296133] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 1643.855598] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1650.191697] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1657.727500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1658.217798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.050386] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 1674.422382] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1679.625604] LustreError: 53016:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1680.465749] Lustre: Modifying parameter general.timeout=1 in log params [ 1682.563758] Lustre: Modifying parameter general.timeout=3 in log params [ 1682.570490] Lustre: Skipped 1 previous similar message [ 1684.681488] Lustre: Modifying parameter general.timeout=5 in log params [ 1684.685449] Lustre: Skipped 1 previous similar message [ 1689.704528] Lustre: Modifying parameter general.timeout=10 in log params [ 1689.708523] Lustre: Skipped 4 previous similar messages [ 1698.937273] Lustre: Modifying parameter general.timeout=19 in log params [ 1698.945963] Lustre: Skipped 8 previous similar messages [ 1715.021579] Lustre: Modifying parameter general.timeout=35 in log params [ 1715.026551] Lustre: Skipped 15 previous similar messages [ 1747.766831] Lustre: Modifying parameter general.timeout=67 in log params [ 1747.773746] Lustre: Skipped 31 previous similar messages [ 1811.847808] Lustre: Modifying parameter general.timeout=126 in log params [ 1811.850754] Lustre: Skipped 58 previous similar messages [ 1829.579468] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:37:26 (1788835046) [ 1833.355468] Lustre: 59983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1835.614251] Lustre: 60082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1838.156077] Lustre: 60180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1842.622500] Lustre: 60390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1842.636850] Lustre: 60390:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1877.786389] Lustre: 61855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1877.793166] Lustre: 61855:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1877.811858] LustreError: 61855:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1884.587522] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:38:20 (1788835100) [ 1902.565555] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.573753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1902.574199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1902.584106] Lustre: Skipped 1 previous similar message [ 1904.945967] Lustre: server umount lustre-MDT0000 complete [ 1907.686994] LustreError: 51509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1907.717980] LustreError: 51509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1908.918301] LustreError: 54061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835128 with bad export cookie 4546249071056473012 [ 1908.929243] LustreError: 54061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1908.937253] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1909.317522] Lustre: server umount lustre-MDT0001 complete [ 1923.670257] Lustre: server umount lustre-OST0000 complete [ 1951.220264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.489697] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1951.509662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1951.514607] Lustre: Skipped 26 previous similar messages [ 1951.666910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1956.332575] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 1967.071159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.195774] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1967.219154] Lustre: 64389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1972.986302] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 1981.417760] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1987.559333] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1994.824946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1995.053876] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1995.475720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.481337] Lustre: Skipped 1 previous similar message [ 2001.919504] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2010.001801] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2012.058596] Lustre: 66957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2022.374379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2022.378503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.383984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2022.383992] Lustre: Skipped 2 previous similar messages [ 2022.398334] Lustre: Skipped 2 previous similar messages [ 2025.864755] Lustre: server umount lustre-OST0000 complete [ 2032.613775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.633561] Lustre: Skipped 1 previous similar message [ 2037.234476] Lustre: server umount lustre-MDT0000 complete [ 2037.731489] LustreError: 64396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.755409] LustreError: 64396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2041.682218] LustreError: 64380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835261 with bad export cookie 4546249071056476302 [ 2041.701954] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2041.716690] LustreError: 64380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2042.131989] Lustre: server umount lustre-MDT0001 complete [ 2059.048752] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [ 2063.244538] Key type lgssc unregistered [ 2063.578642] LNet: 68181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.596092] LNetError: 68181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2064.616573] LNet: Removed LNI 192.168.201.143@tcp [ 2066.283921] Key type .llcrypt unregistered [ 2066.286718] Key type ._llcrypt unregistered [ 2095.305370] Key type ._llcrypt registered [ 2095.306885] Key type .llcrypt registered [ 2095.432133] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 2096.302126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.343446] alg: No test for adler32 (adler32-zlib) [ 2097.605689] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2097.840899] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [ 2099.552241] Key type lgssc registered [ 2100.694728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2109.325042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2109.346834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.877307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.126670] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2120.162588] LustreError: 69607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.183242] LustreError: 69607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2125.079127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.285031] LustreError: 69608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.989991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2130.613987] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2138.106783] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2144.186629] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2151.915525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2152.578361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2159.816737] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2170.124742] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2178.198866] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2191.850995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.855576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2191.871076] Lustre: Skipped 2 previous similar messages [ 2191.874988] Lustre: Skipped 1 previous similar message [ 2196.431716] Lustre: server umount lustre-MDT0000 complete [ 2196.964393] LustreError: 72017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2197.000366] LustreError: 72017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2200.842690] LustreError: 71458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835420 with bad export cookie 7813949541934527244 [ 2200.853319] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2200.861398] LustreError: 71458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2201.409575] Lustre: server umount lustre-MDT0001 complete [ 2215.530784] Lustre: server umount lustre-OST0000 complete [ 2247.721631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2247.981344] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2248.015400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2248.287074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.136452] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2264.850659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.058924] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2265.093232] Lustre: 74482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2265.826639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2271.640750] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2283.174407] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2292.267753] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2301.760872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2302.166795] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2302.752167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.324925] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2321.092863] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2323.524760] Lustre: 77059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2325.674478] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2335.719262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2335.720528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2335.737032] LustreError: Skipped 1 previous similar message [ 2335.739767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2335.739775] Lustre: Skipped 1 previous similar message [ 2335.766752] Lustre: Skipped 1 previous similar message [ 2337.633415] Lustre: server umount lustre-OST0000 complete [ 2342.369928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2342.378966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.407540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2342.414604] Lustre: Skipped 1 previous similar message [ 2345.955123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2345.962471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.796991] Lustre: server umount lustre-MDT0000 complete [ 2351.074809] LustreError: 74492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.100527] LustreError: 74492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2352.662995] LustreError: 77061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835572 with bad export cookie 7813949541934528091 [ 2352.667087] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2352.683485] LustreError: 77061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2353.052383] Lustre: server umount lustre-MDT0001 complete [ 2366.648739] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [ 2369.456798] Key type lgssc unregistered [ 2369.795930] LNet: 78272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.802566] LNetError: 78272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.821298] LNet: Removed LNI 192.168.201.143@tcp [ 2370.813595] Key type .llcrypt unregistered [ 2370.816376] Key type ._llcrypt unregistered [ 2401.256309] Key type ._llcrypt registered [ 2401.259870] Key type .llcrypt registered [ 2401.412301] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 2403.451848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2403.579803] alg: No test for adler32 (adler32-zlib) [ 2404.902864] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2405.281280] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [ 2407.192365] Key type lgssc registered [ 2409.719769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2421.980421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2422.016444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.962575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.372268] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2433.506280] LustreError: 79684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.555268] LustreError: 79684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2438.633567] LustreError: 79685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.430059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.760064] LustreError: 79684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.876111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2450.931763] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2462.654962] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2471.149422] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2480.388777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2481.172347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.441333] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2498.883556] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2506.581602] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2507.536981] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2508.667233] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2510.908475] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2510.916176] Lustre: Skipped 1 previous similar message [ 2515.206617] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2515.213735] Lustre: Skipped 3 previous similar messages [ 2523.515377] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2523.522363] Lustre: Skipped 6 previous similar messages [ 2539.795485] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2539.797878] Lustre: Skipped 15 previous similar messages [ 2573.073759] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2573.079340] Lustre: Skipped 31 previous similar messages [ 2624.545594] Lustre: 87801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2628.199812] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:50:44 (1788835844) [ 2630.764542] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2632.949542] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:50:49 (1788835849) [ 2638.827057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2638.847094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2638.875988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.890560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.891739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.937240] Lustre: Skipped 1 previous similar message [ 2644.630382] Lustre: server umount lustre-OST0000 complete [ 2649.058475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2649.064912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2649.079917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.129505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.130769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.251273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.265098] Lustre: Skipped 2 previous similar messages [ 2663.392242] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2663.733540] Lustre: server umount lustre-MDT0000 complete [ 2666.472399] LustreError: 80417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.495251] LustreError: 80417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2668.842774] LustreError: 79666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835888 with bad export cookie 2802234311176837812 [ 2668.848324] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2668.856266] LustreError: 79666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2669.683455] Lustre: server umount lustre-MDT0001 complete [ 2688.063966] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [ 2690.878148] Key type lgssc unregistered [ 2691.205976] LNet: 89070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2691.218598] LNetError: 89070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2691.256408] LNet: Removed LNI 192.168.201.143@tcp [ 2692.314767] Key type .llcrypt unregistered [ 2692.316979] Key type ._llcrypt unregistered [ 2697.396236] Key type ._llcrypt registered [ 2697.401350] Key type .llcrypt registered [ 2697.541925] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2702.445288] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules [ 2703.842094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2703.858758] alg: No test for adler32 (adler32-zlib) [ 2704.874925] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2705.146585] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [ 2706.888520] Key type lgssc registered [ 2708.177713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2709.291589] LustreError: 90269:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2729.107944] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 2731.798494] LustreError: 90269:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2739.035780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2739.060140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2740.708162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.689098] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2750.955137] LustreError: 91295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.962905] LustreError: 91295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2755.701049] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:52:52 (1788835972) [ 2756.067360] LustreError: 91294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.214863] LustreError: 91295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.843738] LustreError: 91326:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2761.870197] LustreError: 91326:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2761.887260] LustreError: 91326:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2762.507231] Lustre: server umount lustre-MDT0000 complete [ 2778.034125] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [ 2781.603817] Key type lgssc unregistered [ 2781.939807] LNet: 92819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2781.949251] LNetError: 92819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2781.971318] LNet: Removed LNI 192.168.201.143@tcp [ 2783.391722] Key type .llcrypt unregistered [ 2783.394354] Key type ._llcrypt unregistered [ 2808.299860] Key type ._llcrypt registered [ 2808.302628] Key type .llcrypt registered [ 2808.469818] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 2810.032035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2810.138430] alg: No test for adler32 (adler32-zlib) [ 2811.249644] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2811.681583] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [ 2813.392198] Key type lgssc registered [ 2815.075845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2823.976182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2824.021913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.561480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2831.075068] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2835.945792] LustreError: 94034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.969145] LustreError: 94034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2840.638561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.401653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2846.425150] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2855.225669] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2862.389123] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2869.971499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2870.663565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.205435] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2885.553138] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2936.856531] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:55:54 (1788836154) [ 2938.907669] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2940.916253] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:55:57 (1788836157) [ 2946.533482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.545796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2946.559419] Lustre: Skipped 1 previous similar message [ 2946.583113] Lustre: Skipped 2 previous similar messages [ 2951.649696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2951.664059] Lustre: Skipped 2 previous similar messages [ 2952.182811] Lustre: server umount lustre-MDT0000 complete [ 2955.695632] LustreError: 94014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836175 with bad export cookie 11067181058108183699 [ 2955.703308] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2955.713270] LustreError: 94014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2956.206701] Lustre: server umount lustre-MDT0001 complete [ 2970.945132] Lustre: server umount lustre-OST0000 complete [ 2982.599830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.143559] LustreError: 98311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.369440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2988.520207] LustreError: 98312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2989.598959] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 2993.642197] LustreError: 98311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2998.754230] LustreError: 98312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2999.762375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.781414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3006.873891] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3015.309900] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3022.618839] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3038.041823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3038.142100] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3038.147615] LustreError: 98301:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3038.162831] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3038.168827] LustreError: 100397:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 3038.172778] LustreError: 100397:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 3038.178365] LustreError: 100397:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3038.226955] Lustre: server umount lustre-OST0000 complete [ 3038.229630] LustreError: 100397:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3046.899793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3047.044608] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3047.050837] LustreError: 98299:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3047.066475] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3047.073474] LustreError: 100862:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 3047.078554] LustreError: 100862:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 3047.083870] LustreError: 100862:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3047.128697] Lustre: server umount lustre-OST0000 complete [ 3047.132493] LustreError: 100862:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3057.881111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3058.222940] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3058.239847] Lustre: Found index 0 for lustre-OST0000, updating log [ 3058.251612] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3058.372162] Lustre: lustre-OST0000: new disk, initializing [ 3058.387342] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3058.401599] Lustre: 101396:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3058.858279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3067.324132] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3072.436293] Lustre: Failing over lustre-OST0000 [ 3072.799825] Lustre: server umount lustre-OST0000 complete [ 3080.676185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.677563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.701339] Lustre: Skipped 2 previous similar messages [ 3080.711784] Lustre: Skipped 1 previous similar message [ 3084.371370] Lustre: server umount lustre-MDT0000 complete [ 3085.829261] LustreError: 99244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.871474] LustreError: 99244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3090.007854] LustreError: 98292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836309 with bad export cookie 11067181058108186443 [ 3090.014819] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3090.028086] LustreError: 98292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3099.609677] Lustre: server umount lustre-MDT0001 complete [ 3110.618786] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:58:47 (1788836327) [ 3128.628962] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 3139.784968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.103380] LustreError: 103749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.318294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3146.425627] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3157.309470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.549551] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3166.992745] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3174.836472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3175.945192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3175.955804] Lustre: Skipped 1 previous similar message [ 3175.978462] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3183.086792] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3183.123564] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3183.124114] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3183.352510] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3194.272439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3194.382289] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3194.480373] Lustre: lustre-OST0001: new disk, initializing [ 3194.484553] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3194.492108] Lustre: 106067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3200.245779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3200.261372] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3200.384407] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3202.632693] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3212.295831] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3222.612465] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3230.177483] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3230.180703] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.201501] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3233.369555] Lustre: server umount lustre-MDT0001 complete [ 3236.330594] LustreError: 103749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3236.351191] LustreError: 103749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 3243.801361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3243.921405] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3243.934314] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3243.978984] Lustre: 103738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3244.805026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3244.814045] Lustre: Skipped 1 previous similar message [ 3245.183670] LustreError: 107616:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3245.196445] LustreError: 107616:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3245.213433] LustreError: 103741:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3245.324258] LustreError: 107622:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.143@tcp: cfg command failed: rc = -17 [ 3245.330575] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3245.330575] [ 3249.457151] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3250.156596] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3260.676227] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:01:17 (1788836477) [ 3280.907972] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 3300.568531] Lustre: 109694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3330.943562] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3334.589637] Lustre: 110829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3423.718494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3423.752348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.801440] Lustre: Skipped 2 previous similar messages [ 3423.812790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3423.829972] Lustre: Skipped 4 previous similar messages [ 3427.307789] LustreError: 104204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3427.329478] LustreError: 104204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3427.731120] Lustre: server umount lustre-MDT0000 complete [ 3432.964613] LustreError: 103731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836652 with bad export cookie 11067181058108187444 [ 3432.977732] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3432.988181] LustreError: 103731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3433.645349] Lustre: server umount lustre-MDT0001 complete [ 3448.278198] Lustre: server umount lustre-OST0000 complete [ 3463.373535] Lustre: server umount lustre-OST0001 complete [ 3469.803464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3480.585446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3489.978943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3503.670464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3526.759417] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 3536.768334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3536.792508] Lustre: lustre-MDT0000: reset Object Index mappings [ 3537.380518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3541.894398] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3552.229285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3552.247589] Lustre: lustre-MDT0001: reset Object Index mappings [ 3558.647904] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3561.750987] Lustre: 115931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3569.834264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3577.064400] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3577.960822] LustreError: 116285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3577.996108] LustreError: 116285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3578.016629] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3583.111616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3586.614937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3592.714930] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3592.718873] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3593.160192] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3602.429357] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3607.068849] Lustre: 117810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3659.014620] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 23:07:56 (1788836876) [ 3684.833240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3684.844344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3684.856094] Lustre: Skipped 6 previous similar messages [ 3684.870593] Lustre: Skipped 3 previous similar messages [ 3699.171084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3699.459765] Lustre: server umount lustre-MDT0000 complete [ 3703.239641] LustreError: 114763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836923 with bad export cookie 11067181058108234337 [ 3703.241969] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3703.260445] LustreError: 114763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3705.314372] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.320831] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3705.320909] Lustre: Skipped 1 previous similar message [ 3705.331569] Lustre: Skipped 9 previous similar messages [ 3709.825221] Lustre: server umount lustre-MDT0001 complete [ 3720.558812] Lustre: server umount lustre-OST0000 complete [ 3730.885431] Lustre: server umount lustre-OST0001 complete [ 3737.942228] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 3753.024374] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 3796.515041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3796.979982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3797.035836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3797.095946] Lustre: lustre-MDT0000: new disk, initializing [ 3797.181544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3797.187327] Lustre: Skipped 3 previous similar messages [ 3797.206936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3802.138139] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3809.704375] Lustre: server umount lustre-MDT0000 complete [ 3821.326646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3821.563165] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3821.580565] Lustre: 122100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3821.599820] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3826.743886] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3833.089898] Lustre: server umount lustre-MDT0000 complete [ 3844.213832] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:11:00 (1788837060) [ 3846.021674] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3848.083131] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:11:05 (1788837065) [ 3869.151446] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 3881.721363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3887.759743] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3899.643306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3899.780251] Lustre: 124153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3899.843735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3899.847281] Lustre: Skipped 1 previous similar message [ 3899.952272] Lustre: lustre-MDT0001: new disk, initializing [ 3900.145637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3900.179699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3905.174343] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3910.813941] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3918.629434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3919.109240] Lustre: lustre-OST0000: new disk, initializing [ 3919.116665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3919.128148] Lustre: 125730:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3920.903687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3920.931219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3921.057319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3926.885810] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3940.548817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3940.767979] Lustre: lustre-OST0001: new disk, initializing [ 3940.776351] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3940.786850] Lustre: 126603:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3942.621748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3942.631572] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3942.833286] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3949.447617] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 3960.871716] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3965.754942] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4393.141665] Lustre: 133033:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4393.148981] Lustre: 133033:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4393.152575] Lustre: 133033:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4393.162269] Lustre: 133033:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4393.170167] Lustre: 133033:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4393.178431] Lustre: 133033:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4690.964850] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:25:07 (1788837907) [ 4708.840282] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4708.853067] Lustre: Skipped 1 previous similar message [ 4708.865370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4708.878822] Lustre: Skipped 5 previous similar messages [ 4712.876482] Lustre: server umount lustre-MDT0000 complete [ 4713.959713] LustreError: 127413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4713.994581] LustreError: 127413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 4717.638141] LustreError: 133068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837937 with bad export cookie 11067181058108302860 [ 4717.642906] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4717.650514] LustreError: 133068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4719.076517] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.084384] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4719.089478] Lustre: Skipped 3 previous similar messages [ 4724.178660] Lustre: server umount lustre-MDT0001 complete [ 4734.166632] Lustre: server umount lustre-OST0000 complete [ 4744.692891] Lustre: server umount lustre-OST0001 complete [ 4752.488980] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 4764.612101] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 4803.660858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4803.901770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4803.926308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4804.027046] Lustre: lustre-MDT0000: new disk, initializing [ 4804.157077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.167792] Lustre: Skipped 5 previous similar messages [ 4804.193342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4808.820724] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 4819.870677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4820.018849] Lustre: 136087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4820.135166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4820.143521] Lustre: Skipped 1 previous similar message [ 4820.307705] Lustre: lustre-MDT0001: new disk, initializing [ 4820.498434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4820.520373] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4825.429598] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 4835.550977] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4841.153867] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4848.767443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4849.216318] Lustre: lustre-OST0000: new disk, initializing [ 4849.219629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4849.232909] Lustre: 138054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4850.384668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4850.409518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4850.614100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4856.608970] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 4867.492281] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4879.224516] Lustre: lustre-MDD0000: changelog on [ 4882.421930] Lustre: lustre-MDD0001: changelog on [ 6929.520880] Lustre: lustre-MDD0001: changelog off [ 6931.768650] Lustre: 142634:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6932.733467] Lustre: lustre-MDD0000: changelog off [ 6939.616979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6939.635340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6939.653575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6939.657470] Lustre: Skipped 1 previous similar message [ 6939.671702] Lustre: Skipped 4 previous similar messages [ 6944.761735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6944.774914] Lustre: Skipped 1 previous similar message [ 6945.325044] Lustre: server umount lustre-OST0000 complete [ 6950.371069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6950.378663] LustreError: Skipped 1 previous similar message [ 6950.379896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.390133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6954.981322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6954.981993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6955.013280] Lustre: Skipped 1 previous similar message [ 6956.721552] Lustre: server umount lustre-MDT0000 complete [ 6960.100713] LustreError: 136099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6960.134704] LustreError: 136099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6961.992757] LustreError: 136080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840181 with bad export cookie 11067181058109130561 [ 6962.002047] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6962.019782] LustreError: 136080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6963.126209] Lustre: server umount lustre-MDT0001 complete [ 6978.003340] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [ 6980.732793] Key type lgssc unregistered [ 6981.033506] LNet: 147230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6981.046526] LNetError: 147230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6981.070310] LNet: Removed LNI 192.168.201.143@tcp [ 6982.146174] Key type .llcrypt unregistered [ 6982.153405] Key type ._llcrypt unregistered [ 7003.690667] Key type ._llcrypt registered [ 7003.694926] Key type .llcrypt registered [ 7003.827788] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 7021.676958] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 7023.030930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7023.078670] alg: No test for adler32 (adler32-zlib) [ 7024.118212] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 7024.292296] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [ 7025.960184] Key type lgssc registered [ 7027.100580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7067.716718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7067.769920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.205237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7069.251683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7069.321133] Lustre: lustre-MDT0000: new disk, initializing [ 7069.441815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7069.454181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7074.998626] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7089.113909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7089.216291] Lustre: 150427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7089.254121] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7089.256993] Lustre: Skipped 1 previous similar message [ 7089.327841] Lustre: lustre-MDT0001: new disk, initializing [ 7089.425255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7089.482386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7089.504470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7094.630328] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7104.581429] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7112.546068] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7121.748299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7122.122533] Lustre: lustre-OST0000: new disk, initializing [ 7122.127556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7122.136397] Lustre: 152395:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7122.224477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7128.410568] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7129.202478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7129.218519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7129.311062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7141.136960] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7148.475417] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_module obdecho/obdecho [ 7151.205635] LustreError: 153398:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7151.214690] LustreError: 153398:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7153.369749] LustreError: 153495:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7170.540094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7170.550423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7170.558571] Lustre: Skipped 2 previous similar messages [ 7170.577803] Lustre: Skipped 2 previous similar messages [ 7173.809126] Lustre: server umount lustre-MDT0000 complete [ 7177.976381] LustreError: 150421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840397 with bad export cookie 14967895789894853843 [ 7177.996316] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7178.271775] Lustre: server umount lustre-MDT0001 complete [ 7193.821049] Lustre: server umount lustre-OST0000 complete [ 7217.628645] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 7229.544890] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 7272.231599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7272.607947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7272.694311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7272.848365] Lustre: lustre-MDT0000: new disk, initializing [ 7272.969558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7273.002668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7277.888382] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7291.758138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7291.899223] Lustre: 157212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7291.944386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7291.947308] Lustre: Skipped 1 previous similar message [ 7292.014885] Lustre: lustre-MDT0001: new disk, initializing [ 7292.117704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7292.161450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7292.180348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7297.022333] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7308.190717] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7316.113528] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7325.302432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7325.706586] Lustre: lustre-OST0000: new disk, initializing [ 7325.710633] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7325.723906] Lustre: 159184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7325.848471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7330.901758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7330.913935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7331.030067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7334.130312] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7345.963903] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7351.457598] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7351.668823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7357.144396] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7357.450657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7362.030682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.035084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7362.035804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7362.045637] Lustre: Skipped 1 previous similar message [ 7367.153187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7367.163566] Lustre: Skipped 2 previous similar messages [ 7372.258608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7372.263690] Lustre: Skipped 1 previous similar message [ 7374.308979] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7374.682899] Lustre: server umount lustre-OST0000 complete [ 7382.501775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7382.508224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7382.543448] Lustre: Skipped 1 previous similar message [ 7382.567291] Lustre: Skipped 1 previous similar message [ 7385.928108] Lustre: server umount lustre-MDT0000 complete [ 7387.618289] LustreError: 157219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7387.640768] LustreError: 157219:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7390.793875] LustreError: 157204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840610 with bad export cookie 14967895789894854802 [ 7390.800968] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7390.819749] LustreError: 157204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7391.510730] Lustre: server umount lustre-MDT0001 complete [ 7397.483675] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 00:10:13 (1788840613) [ 7400.454801] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7403.514407] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 00:10:19 (1788840619) [ 7412.880141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7413.554880] LustreError: 161598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7413.713891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.768818] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7418.850811] LustreError: 161599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7423.972944] LustreError: 161598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7427.366345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7432.955985] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7441.940845] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7448.810880] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7457.463593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7458.168151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7458.199031] Lustre: Skipped 1 previous similar message [ 7465.798871] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7473.520052] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7479.673453] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.43@tcp (stopping) [ 7483.879249] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7483.882292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7483.902260] Lustre: Skipped 1 previous similar message [ 7483.924175] LustreError: Skipped 1 previous similar message [ 7485.342095] Lustre: server umount lustre-MDT0000 complete [ 7488.815996] LustreError: 161580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840708 with bad export cookie 14967895789894855684 [ 7488.826360] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7488.999627] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7489.001518] LustreError: 161594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7489.012425] Lustre: Skipped 1 previous similar message [ 7498.215900] LustreError: 161595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7498.219378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7498.242663] LustreError: 161595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7498.248429] Lustre: Skipped 8 previous similar messages [ 7503.328922] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7503.544944] Lustre: server umount lustre-MDT0001 complete [ 7509.629967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7522.452226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7522.670437] LustreError: 165441:0:(llog.c:1655:llog_backup()) MGC192.168.201.143@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7522.679333] Lustre: 165441:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.143@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7529.814673] LustreError: 165446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7529.848972] LustreError: 165446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7563.399264] LustreError: 163430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7563.428435] LustreError: 163430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [ 7583.451885] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7583.454745] Lustre: 165441:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.143@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7624.179482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7629.281917] LustreError: 165448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7629.295743] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7629.326711] LustreError: 165448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7629.966700] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7640.726056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7658.479764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7663.539238] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7664.119301] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7672.770689] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7674.748616] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7681.924564] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7683.935756] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7705.064950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7705.071297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.072957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7705.089414] Lustre: Skipped 2 previous similar messages [ 7705.106535] Lustre: Skipped 4 previous similar messages [ 7708.688360] Lustre: server umount lustre-MDT0000 complete [ 7713.592249] LustreError: 161578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840933 with bad export cookie 14967895789894862145 [ 7713.601976] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7713.613127] LustreError: 161578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7714.253702] Lustre: server umount lustre-MDT0001 complete [ 7730.021159] Lustre: server umount lustre-OST0000 complete [ 7758.577037] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 7772.278350] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 7815.678443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7815.994947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7816.041451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7816.121178] Lustre: lustre-MDT0000: new disk, initializing [ 7816.190928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7816.194047] Lustre: Skipped 1 previous similar message [ 7816.208156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7822.841074] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7835.073399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.302742] Lustre: 170554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7835.791322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7840.946584] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7854.046408] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7861.004592] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7868.019904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7868.485487] Lustre: lustre-OST0000: new disk, initializing [ 7868.492340] Lustre: Skipped 1 previous similar message [ 7868.497882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7868.504494] Lustre: Skipped 2 previous similar messages [ 7868.516792] Lustre: 172524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7875.153248] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7876.229444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7876.244275] Lustre: Skipped 1 previous similar message [ 7876.248560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7876.379162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7885.756379] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7890.679885] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7890.996208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7896.405667] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7896.659858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7902.181267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7902.182177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7902.199990] LustreError: Skipped 1 previous similar message [ 7902.219197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7902.232744] Lustre: Skipped 1 previous similar message [ 7905.315368] Lustre: server umount lustre-OST0000 complete [ 7912.424221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7924.192494] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7924.484950] Lustre: server umount lustre-MDT0000 complete [ 7927.783851] LustreError: 170567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7927.803830] LustreError: 170567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [ 7929.135585] LustreError: 170546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841149 with bad export cookie 14967895789894878966 [ 7929.148887] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7929.166676] LustreError: 170546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7929.641600] Lustre: server umount lustre-MDT0001 complete [ 7940.961829] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:19:17 (1788841157) [ 7950.418153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7956.354444] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7967.013756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7973.169308] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 7981.931824] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7989.047273] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7998.105408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8006.203945] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8015.921542] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8034.216541] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:20:50 (1788841250) [ 8043.489615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8043.501063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8043.526489] Lustre: Skipped 3 previous similar messages [ 8043.551103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8043.554045] Lustre: Skipped 6 previous similar messages [ 8048.067187] Lustre: server umount lustre-OST0000 complete [ 8058.927136] Lustre: server umount lustre-MDT0000 complete [ 8063.351779] LustreError: 174920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841283 with bad export cookie 14967895789894879855 [ 8063.375184] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8063.853198] Lustre: server umount lustre-MDT0001 complete [ 8077.481659] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [ 8080.729261] Key type lgssc unregistered [ 8081.129701] LNet: 178682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8081.146778] LNetError: 178682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8082.216826] LNet: Removed LNI 192.168.201.143@tcp [ 8083.442834] Key type .llcrypt unregistered [ 8083.446402] Key type ._llcrypt unregistered [ 8106.048293] Key type ._llcrypt registered [ 8106.053915] Key type .llcrypt registered [ 8106.182823] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 8107.334553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8107.531221] alg: No test for adler32 (adler32-zlib) [ 8108.808444] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 8109.075043] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [ 8110.784181] Key type lgssc registered [ 8112.470117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8120.889622] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8120.920359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8136.481048] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8141.600447] LustreError: 179923:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.143@tcp: failed processing log, type 4: rc = -110 [ 8146.720946] LustreError: 179923:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8146.742884] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8146.753208] LustreError: MGC192.168.201.143@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8146.766539] LustreError: 179923:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8146.781582] LustreError: 179923:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 8146.786372] LustreError: 179923:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8146.830444] Lustre: server umount lustre-OST0000 complete [ 8146.837916] LustreError: 179923:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8157.629722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8159.227542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8166.140857] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8169.453328] LustreError: 180421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8169.478245] LustreError: 180421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8174.564864] LustreError: 180422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8179.689586] LustreError: 180421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8179.984657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.973236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8186.658587] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8196.716565] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8203.044560] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8210.045525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8210.235892] LustreError: 182241:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8210.241578] Lustre: 182241:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.143@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8210.461597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8217.953620] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8227.211887] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8229.885071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8229.889425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8229.927130] Lustre: Skipped 2 previous similar messages [ 8229.954008] Lustre: Skipped 2 previous similar messages [ 8234.980398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8234.993033] Lustre: Skipped 2 previous similar messages [ 8236.343744] Lustre: server umount lustre-MDT0000 complete [ 8240.124177] LustreError: 181150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8240.155934] LustreError: 181150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8240.818671] LustreError: 180402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841460 with bad export cookie 5118913734242613227 [ 8240.829183] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8241.208379] LustreError: 181152:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005acf425b x1875736084379008/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8241.492917] Lustre: server umount lustre-MDT0001 complete [ 8257.307243] Lustre: server umount lustre-OST0000 complete [ 8270.022615] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 8288.256292] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 8331.081689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8331.304921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8331.336211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8331.446280] Lustre: lustre-MDT0000: new disk, initializing [ 8331.569278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8331.603385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8337.711124] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8349.643732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8349.820106] Lustre: 185787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8349.876460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8349.879824] Lustre: Skipped 1 previous similar message [ 8349.952770] Lustre: lustre-MDT0001: new disk, initializing [ 8350.102284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8350.146546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8350.170075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8356.346192] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8369.167152] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8376.245621] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8385.580588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8385.887422] Lustre: lustre-OST0000: new disk, initializing [ 8385.894119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8385.902893] Lustre: 187757:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8386.060532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8392.312282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8392.321435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8392.443185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8393.645381] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8405.489991] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8411.325883] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8411.593617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8416.691504] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8416.955978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8423.394393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8423.395280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8423.409944] LustreError: Skipped 1 previous similar message [ 8423.410625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8423.436512] Lustre: Skipped 1 previous similar message [ 8428.521531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8428.529900] Lustre: Skipped 2 previous similar messages [ 8433.120337] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8433.326504] Lustre: server umount lustre-OST0000 complete [ 8438.754242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8438.778602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8438.788976] Lustre: Skipped 1 previous similar message [ 8444.200428] Lustre: server umount lustre-MDT0000 complete [ 8448.368936] LustreError: 185781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841668 with bad export cookie 5118913734242613941 [ 8448.374544] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8448.383444] LustreError: 185781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8448.809645] Lustre: server umount lustre-MDT0001 complete [ 8459.053557] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:27:55 (1788841675) [ 8478.642834] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [ 8482.756302] Key type lgssc unregistered [ 8482.996040] LNet: 190703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8483.005976] LNetError: 190703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8483.042452] LNet: Removed LNI 192.168.201.143@tcp [ 8484.310753] Key type .llcrypt unregistered [ 8484.316670] Key type ._llcrypt unregistered [ 8508.670243] Key type ._llcrypt registered [ 8508.672504] Key type .llcrypt registered [ 8508.807989] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 8509.886861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8509.962526] alg: No test for adler32 (adler32-zlib) [ 8511.381975] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 8511.764662] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [ 8513.512167] Key type lgssc registered [ 8515.026795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8524.617556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8524.644522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8526.082390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8532.021516] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8535.528621] LustreError: 191925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8535.545141] LustreError: 191925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8540.645350] LustreError: 191926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8541.277459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8541.647406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8546.990369] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8555.892174] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8562.599785] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8571.028488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8571.238081] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8571.241755] LustreError: 193789:0:(llog.c:1696:llog_backup()) MGC192.168.201.143@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8571.259938] Lustre: 193789:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.143@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8571.408772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8577.448785] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8586.148456] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8588.651493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8588.659707] Lustre: Skipped 1 previous similar message [ 8588.669138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8588.675641] Lustre: Skipped 2 previous similar messages [ 8593.889991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8593.904384] Lustre: Skipped 2 previous similar messages [ 8594.967910] Lustre: server umount lustre-MDT0000 complete [ 8599.011790] LustreError: 192653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8599.044981] LustreError: 192653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8599.339449] LustreError: 191905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841819 with bad export cookie 9682096116631820380 [ 8599.353885] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8599.371537] LustreError: 191905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8599.792392] Lustre: server umount lustre-MDT0001 complete [ 8614.622799] Lustre: server umount lustre-OST0000 complete [ 8624.872369] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 8637.921240] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 8679.725560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8680.013795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8680.035898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8680.087193] Lustre: lustre-MDT0000: new disk, initializing [ 8680.144880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8680.154336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8685.624312] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8697.950120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8698.085703] Lustre: 197316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8698.122948] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8698.125460] Lustre: Skipped 1 previous similar message [ 8698.127704] LustreError: 198100:0:(llog.c:1696:llog_backup()) MGC192.168.201.143@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8698.132819] LustreError: 198100:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8698.138231] Lustre: 198100:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.143@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8698.161286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8698.166033] Lustre: Skipped 1 previous similar message [ 8698.241546] Lustre: lustre-MDT0001: new disk, initializing [ 8698.359265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8698.406090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8698.433709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8703.231775] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8714.293367] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8720.431241] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8729.351827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8729.724634] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8729.727279] Lustre: Skipped 1 previous similar message [ 8729.730386] LustreError: 199263:0:(llog.c:1696:llog_backup()) MGC192.168.201.143@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8729.743694] LustreError: 199263:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8729.746841] Lustre: 199263:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.143@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8729.760143] Lustre: 199263:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8729.788344] Lustre: lustre-OST0000: new disk, initializing [ 8729.793872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8729.811796] Lustre: 199286:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8729.947068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.659244] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8738.405238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8738.429322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8738.534021] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8748.073847] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8753.676390] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8753.923271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8758.900460] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8759.144440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8764.388464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8764.400049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8764.411401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8764.437422] Lustre: Skipped 2 previous similar messages [ 8767.929468] Lustre: server umount lustre-OST0000 complete [ 8774.626152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8774.646405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8774.647096] Lustre: Skipped 1 previous similar message [ 8774.654869] Lustre: Skipped 2 previous similar messages [ 8778.664958] Lustre: server umount lustre-MDT0000 complete [ 8779.781593] LustreError: 198953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8779.811199] LustreError: 198953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8784.717867] LustreError: 199285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842004 with bad export cookie 9682096116631821094 [ 8784.718314] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8784.729208] LustreError: 199285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8785.306322] Lustre: server umount lustre-MDT0001 complete [ 8797.453759] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:33:34 (1788842014) [ 8821.233389] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 8836.644405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8837.205985] LustreError: 202437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8837.266864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8843.352487] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8847.842761] LustreError: 202438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8847.873864] LustreError: 202438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8854.191731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8860.046686] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8863.152716] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8870.859706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8871.338060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8871.349120] Lustre: Skipped 1 previous similar message [ 8879.064959] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8889.441207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8889.679649] Lustre: lustre-OST0001: new disk, initializing [ 8889.688187] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8889.697511] Lustre: 204744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8896.477410] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8898.161660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8898.174546] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8898.308616] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8908.815082] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8918.894692] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8921.533221] Lustre: DEBUG MARKER: ADD OST3 [ 8927.754205] loop: module loaded [ 8927.911080] loop0: detected capacity change from 0 to 204800000 [ 8928.249487] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8928.272402] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8928.284854] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8935.531319] loop0: detected capacity change from 0 to 204800000 [ 8936.059676] Lustre: DEBUG MARKER: STOP OST3 [ 8943.373531] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8943.748743] Lustre: lustre-OST0003: new disk, initializing [ 8943.756939] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8943.765232] Lustre: 206784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8943.938491] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8943.945897] Lustre: Skipped 1 previous similar message [ 8945.457615] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8945.648575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8945.669033] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8945.724563] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8947.488424] LustreError: 206909:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8948.035969] Lustre: Failing over lustre-OST0003 [ 8948.335577] Lustre: server umount lustre-OST0003 complete [ 8950.110907] LustreError: 203922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8950.121511] LustreError: 203922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8950.772292] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8950.799885] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8953.889552] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 8965.125590] Lustre: DEBUG MARKER: START OST3 again [ 8973.563289] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8973.851822] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8975.144194] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8975.357728] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8975.360035] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8981.006923] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9003.286578] Lustre: Failing over lustre-OST0003 [ 9003.378386] Lustre: server umount lustre-OST0003 complete [ 9004.556668] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9004.579192] Lustre: Skipped 1 previous similar message [ 9004.584028] LustreError: 203923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9004.607491] LustreError: 203923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 9023.458829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9023.463338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9023.475030] Lustre: Skipped 1 previous similar message [ 9023.481743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9035.237838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9035.256545] Lustre: Skipped 11 previous similar messages [ 9036.256140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9036.504619] Lustre: server umount lustre-MDT0000 complete [ 9040.668653] LustreError: 202417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842260 with bad export cookie 9682096116631821976 [ 9040.675969] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9040.690410] LustreError: 202417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9045.474298] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9045.484327] Lustre: Skipped 4 previous similar messages [ 9051.618893] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9051.623630] Lustre: Skipped 5 previous similar messages [ 9055.201327] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9055.603144] Lustre: server umount lustre-MDT0001 complete [ 9059.595159] Lustre: server umount lustre-OST0000 complete [ 9063.246987] Lustre: server umount lustre-OST0001 complete [ 9071.043510] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 9080.762467] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 9123.697030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9124.000472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9124.030275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9124.106277] Lustre: lustre-MDT0000: new disk, initializing [ 9124.180553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9124.185639] Lustre: Skipped 1 previous similar message [ 9124.202287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9129.527300] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9141.515801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9141.675895] Lustre: 212218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9141.771782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9141.781613] Lustre: Skipped 1 previous similar message [ 9141.973398] Lustre: lustre-MDT0001: new disk, initializing [ 9142.268624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9142.280249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9147.441143] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9158.498918] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9164.964872] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9173.070527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9173.546431] Lustre: 214190:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9175.634076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9175.642950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9175.767231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9180.776354] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9191.803401] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9198.215817] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9198.415448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9203.990593] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9204.202604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9206.756372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9206.764617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9206.778155] LustreError: Skipped 1 previous similar message [ 9206.783389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9206.796900] Lustre: Skipped 1 previous similar message [ 9206.815988] Lustre: Skipped 1 previous similar message [ 9212.519610] Lustre: server umount lustre-OST0000 complete [ 9223.114609] Lustre: server umount lustre-MDT0000 complete [ 9227.246753] LustreError: 213065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9227.267338] LustreError: 213065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [ 9228.221205] LustreError: 214189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842448 with bad export cookie 9682096116631824272 [ 9228.228075] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9228.234180] LustreError: 214189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9228.784465] Lustre: server umount lustre-MDT0001 complete [ 9232.781827] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:40:49 (1788842449) [ 9250.486906] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 9263.028151] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 9309.928813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9310.282394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9310.323387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9310.332732] Lustre: Skipped 1 previous similar message [ 9310.455093] Lustre: lustre-MDT0000: new disk, initializing [ 9310.458281] Lustre: Skipped 1 previous similar message [ 9310.586255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9314.889733] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9327.609064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9327.838422] Lustre: 218802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9328.250762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9333.634292] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9345.551292] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9352.607775] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9361.302217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9361.579175] Lustre: 220773:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9363.052493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9363.195989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9369.317597] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9380.442216] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9386.049847] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9386.425738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9391.676924] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9392.014498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9394.146815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9394.161737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9394.169247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9394.189245] Lustre: Skipped 3 previous similar messages [ 9394.212902] Lustre: Skipped 7 previous similar messages [ 9400.335926] Lustre: server umount lustre-OST0000 complete [ 9404.899132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9404.905303] LustreError: Skipped 1 previous similar message [ 9411.151572] Lustre: server umount lustre-MDT0000 complete [ 9414.629969] LustreError: 218814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9414.647628] LustreError: 218814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9415.206287] LustreError: 218795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842635 with bad export cookie 9682096116631825154 [ 9415.213892] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9415.217284] LustreError: 218795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9415.505672] Lustre: server umount lustre-MDT0001 complete [ 9424.679274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9425.602331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9425.606765] Lustre: Skipped 5 previous similar messages [ 9432.170145] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9444.221943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.374786] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9458.631425] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9465.026453] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9473.070327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9480.415464] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9489.182995] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9506.290677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9511.321686] Lustre: server umount lustre-OST0000 complete [ 9521.778929] Lustre: server umount lustre-MDT0000 complete [ 9525.676353] LustreError: 222972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842745 with bad export cookie 9682096116631826029 [ 9525.683586] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9525.686347] LustreError: 222972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9526.036247] Lustre: server umount lustre-MDT0001 complete [ 9628.468791] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [ 9631.737129] Key type lgssc unregistered [ 9632.241228] LNet: 226678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9632.249033] LNetError: 226678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9632.303267] LNet: Removed LNI 192.168.201.143@tcp [ 9633.182160] Key type .llcrypt unregistered [ 9633.188848] Key type ._llcrypt unregistered [ 9663.508647] Key type ._llcrypt registered [ 9663.511795] Key type .llcrypt registered [ 9663.607601] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [ 9680.816664] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 9682.305826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9682.324875] alg: No test for adler32 (adler32-zlib) [ 9683.734420] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 9684.036923] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [ 9685.712255] Key type lgssc registered [ 9686.896751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9728.851655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9728.882100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9730.169592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9730.265802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9730.337630] Lustre: lustre-MDT0000: new disk, initializing [ 9730.417178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9730.440933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9735.306883] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9747.934376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9748.074791] Lustre: 230031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9748.139498] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9748.143531] Lustre: Skipped 1 previous similar message [ 9748.250543] Lustre: lustre-MDT0001: new disk, initializing [ 9748.406221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9748.450593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9748.457638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9753.622726] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9763.780873] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9770.966934] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9778.787358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9779.128601] Lustre: lustre-OST0000: new disk, initializing [ 9779.135732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9779.150057] Lustre: 232001:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9779.345233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9786.581952] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9789.091834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9789.102292] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9789.262256] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9799.226696] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9806.038642] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9806.249184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9812.443739] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9812.852352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9820.144101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9820.156211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9820.163985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9820.171000] Lustre: Skipped 1 previous similar message [ 9820.185116] Lustre: Skipped 1 previous similar message [ 9822.010055] Lustre: server umount lustre-OST0000 complete [ 9830.370322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.376680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9830.395414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9830.409440] LustreError: Skipped 1 previous similar message [ 9830.441989] Lustre: Skipped 1 previous similar message [ 9832.892984] Lustre: server umount lustre-MDT0000 complete [ 9837.199808] LustreError: 230024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843057 with bad export cookie 9322140057870762640 [ 9837.200924] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9837.206640] LustreError: 230024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9837.528361] Lustre: server umount lustre-MDT0001 complete [ 9864.516823] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [ 9877.276449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9877.943846] LustreError: 234962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9877.960265] LustreError: 234962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9877.989358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9882.641645] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9883.111868] LustreError: 234963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9887.213780] LustreError: 234962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9891.791138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9892.375764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9898.054048] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9902.175593] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9910.867111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9911.544879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9919.953266] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9930.536954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9930.807355] Lustre: lustre-OST0001: new disk, initializing [ 9930.810706] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9930.819388] Lustre: 237270:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9930.933850] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9938.512209] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [ 9940.109850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9940.121199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9940.239577] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9950.319255] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9959.712756] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9973.110487] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:53:09 (1788843189) [ 9976.292361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9976.305580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9976.314015] Lustre: Skipped 3 previous similar messages [ 9976.325205] Lustre: Skipped 3 previous similar messages [ 9981.410211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9982.873163] Lustre: server umount lustre-MDT0000 complete [ 9986.534118] LustreError: 234957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9986.552176] LustreError: 234957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9987.949207] LustreError: 238040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843207 with bad export cookie 9322140057870763522 [ 9987.971046] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9988.590403] Lustre: server umount lustre-MDT0001 complete [10004.746817] Lustre: server umount lustre-OST0000 complete [10008.032153] Lustre: 228254:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843211/real 1788843211] req@ffff94ff02d49880 x1875737735973376/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10008.066686] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10008.080492] Lustre: Skipped 1 previous similar message [10009.056251] Lustre: 228257:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843212/real 1788843212] req@ffff9500072b3800 x1875737735973632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843228 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10009.399236] Lustre: server umount lustre-OST0001 complete [10017.629892] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [10030.433384] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [10073.202306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10073.444156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10073.469557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10073.538573] Lustre: lustre-MDT0000: new disk, initializing [10073.588768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10073.601601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10079.069395] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10091.124854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.311868] Lustre: 241234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10091.389335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10091.400614] Lustre: Skipped 1 previous similar message [10091.497822] Lustre: lustre-MDT0001: new disk, initializing [10091.768216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10091.787645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10096.629048] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10108.226358] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10115.431220] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10123.330700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10123.688380] Lustre: lustre-OST0000: new disk, initializing [10123.690140] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10123.695338] Lustre: 243202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10124.881592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10124.896861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10125.082424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10129.759599] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10141.188126] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10146.883298] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10147.138605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10152.235946] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10152.428293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10156.007395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10156.007851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10156.023255] LustreError: Skipped 1 previous similar message [10156.028531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10156.039427] Lustre: Skipped 1 previous similar message [10156.058841] Lustre: Skipped 4 previous similar messages [10160.751700] Lustre: server umount lustre-OST0000 complete [10168.290493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10168.304904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10168.323236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10168.333163] Lustre: Skipped 2 previous similar messages [10171.662902] Lustre: server umount lustre-MDT0000 complete [10174.443786] LustreError: 243275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10174.483754] LustreError: 243275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10177.042096] LustreError: 241227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843396 with bad export cookie 9322140057870765657 [10177.044719] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10177.059923] LustreError: 241227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10177.917320] Lustre: server umount lustre-MDT0001 complete [10187.641719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10187.946634] LustreError: 245420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10188.037181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10188.046678] Lustre: Skipped 2 previous similar messages [10192.586042] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10201.727215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10206.948445] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10216.168623] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10223.049200] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10231.990824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10239.420280] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10248.097605] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10397.666633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10397.693747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10397.741070] Lustre: Skipped 1 previous similar message [10397.761615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10397.777419] Lustre: Skipped 2 previous similar messages [10403.696987] Lustre: server umount lustre-OST0000 complete [10414.935901] Lustre: server umount lustre-MDT0000 complete [10416.100518] LustreError: 245421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10416.141730] LustreError: 245421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10419.831179] LustreError: 245402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843639 with bad export cookie 9322140057870766539 [10419.838726] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10420.467878] Lustre: server umount lustre-MDT0001 complete [10518.164931] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [10522.287241] Key type lgssc unregistered [10522.791727] LNet: 249048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10522.801679] LNetError: 249048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10522.824898] LNet: Removed LNI 192.168.201.143@tcp [10523.967252] Key type .llcrypt unregistered [10523.971736] Key type ._llcrypt unregistered [10557.172967] Key type ._llcrypt registered [10557.176112] Key type .llcrypt registered [10557.439733] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [10573.785847] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [10574.687094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10574.814230] alg: No test for adler32 (adler32-zlib) [10575.881973] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [10576.089929] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [10577.776399] Key type lgssc registered [10579.051196] Lustre: Echo OBD driver; http://www.lustre.org/ [10629.945550] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [10641.663388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10641.680730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10643.040832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10643.076631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10643.143416] Lustre: lustre-MDT0000: new disk, initializing [10643.213955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10643.237418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10648.012982] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10658.537440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10658.691545] Lustre: 253117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10658.788623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10658.804338] Lustre: Skipped 1 previous similar message [10658.834566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10658.935861] Lustre: lustre-MDT0001: new disk, initializing [10659.045584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10659.086432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10659.100697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10664.912442] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10670.995702] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10678.280384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10678.630485] Lustre: lustre-OST0000: new disk, initializing [10678.636327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10678.644482] Lustre: 254748:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10678.741631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10683.527237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10683.549980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10683.747755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10686.646956] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10700.131420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10700.392401] Lustre: lustre-OST0001: new disk, initializing [10700.396752] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10700.404955] Lustre: 255619:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10700.552864] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10705.553652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10705.565565] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10705.712596] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10708.191736] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10720.265135] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10728.591262] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10746.337768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10746.352242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10746.367548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10746.877175] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10746.897982] Lustre: Skipped 1 previous similar message [10748.554213] Lustre: server umount lustre-MDT0000 complete [10752.770210] LustreError: 256389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843972 with bad export cookie 8202569861141870600 [10752.784481] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10752.786062] LustreError: 256389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10753.353804] Lustre: server umount lustre-MDT0001 complete [10767.555929] Lustre: server umount lustre-OST0000 complete [10781.739505] Lustre: server umount lustre-OST0001 complete [10797.081992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10808.102477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10818.338171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10830.397278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10853.469562] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [10864.416779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10864.436131] Lustre: lustre-MDT0000: reset Object Index mappings [10864.731454] LustreError: 260636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10864.747563] LustreError: 260636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10864.801380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10869.398785] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10870.242373] LustreError: 260637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.345285] LustreError: 260636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10877.882859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10877.910171] Lustre: lustre-MDT0001: reset Object Index mappings [10878.484983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10883.180135] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10886.357234] Lustre: 261778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10893.554666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10900.943752] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10903.217188] LustreError: 262132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10908.647106] LustreError: 262133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10909.680606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10910.992939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10911.308320] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10911.311530] Lustre: Skipped 1 previous similar message [10916.399061] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10919.169994] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [10929.066929] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10933.689963] Lustre: 263650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10939.876154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10939.883755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10939.912252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10939.926465] Lustre: Skipped 3 previous similar messages [10941.055719] Lustre: server umount lustre-MDT0000 complete [10941.947920] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10941.951910] LustreError: 261369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10941.976459] Lustre: Skipped 1 previous similar message [10942.022653] LustreError: 261369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10945.421073] LustreError: 261380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844165 with bad export cookie 8202569861141877390 [10945.424571] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10945.448708] LustreError: 261380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10946.314831] Lustre: server umount lustre-MDT0001 complete [10960.365141] Lustre: server umount lustre-OST0000 complete [10962.340788] Lustre: 250622:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844166/real 1788844166] req@ffff94ff02312a00 x1875738672367616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788844182 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10962.384520] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10962.404765] Lustre: Skipped 1 previous similar message [10965.488728] Lustre: server umount lustre-OST0001 complete [10984.796711] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:10:01 (1788844201) [11002.039594] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [11013.607687] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [11050.768305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11051.135342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11051.168862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11051.251234] Lustre: lustre-MDT0000: new disk, initializing [11051.331937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11051.358355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11056.079955] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11066.874671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11067.073420] Lustre: 267901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11067.138320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11067.140890] Lustre: Skipped 1 previous similar message [11067.196381] Lustre: lustre-MDT0001: new disk, initializing [11067.277863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11067.285878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11071.997095] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11080.975301] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11087.756257] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11094.669802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11094.921046] Lustre: lustre-OST0000: new disk, initializing [11094.925428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11094.933280] Lustre: 269872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11096.723336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11096.735966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11096.927925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11101.661186] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11114.006454] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11119.671187] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11119.959620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11125.993599] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11126.321305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11127.778142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11127.798627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11127.807267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11132.905928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11132.914802] Lustre: Skipped 1 previous similar message [11142.112148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11142.315979] Lustre: server umount lustre-OST0000 complete [11148.259109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11148.265350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11148.278989] Lustre: Skipped 2 previous similar messages [11148.283594] Lustre: Skipped 3 previous similar messages [11152.896043] Lustre: server umount lustre-MDT0000 complete [11153.390761] LustreError: 267908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11153.414262] LustreError: 267908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11157.150111] LustreError: 267896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844377 with bad export cookie 8202569861141879483 [11157.153980] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11157.163348] LustreError: 267896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11157.616975] Lustre: server umount lustre-MDT0001 complete [11179.957484] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [11191.381105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11191.898467] LustreError: 272828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11191.993810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11191.999286] Lustre: Skipped 2 previous similar messages [11196.848247] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11205.410685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11211.885197] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11215.506091] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11222.499747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11229.692439] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11238.910617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11239.149886] Lustre: lustre-OST0001: new disk, initializing [11239.154858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11239.160925] Lustre: 275139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11240.451398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11240.469566] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11240.587219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11246.192406] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11257.562295] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11261.122614] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11266.255952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11266.364201] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11276.265584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11276.276798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11276.294749] Lustre: Skipped 2 previous similar messages [11276.312732] Lustre: Skipped 3 previous similar messages [11280.235586] Lustre: server umount lustre-MDT0000 complete [11281.380828] LustreError: 273557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11281.414825] LustreError: 273557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11284.599825] LustreError: 272809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844504 with bad export cookie 8202569861141880365 [11284.610406] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11284.615428] LustreError: 272809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11285.256589] Lustre: server umount lustre-MDT0001 complete [11300.195915] Lustre: server umount lustre-OST0000 complete [11302.690227] Lustre: 250622:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844506/real 1788844506] req@ffff94ff08ec9c00 x1875738672482304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788844522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11304.699375] Lustre: server umount lustre-OST0001 complete [11324.085223] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [11334.782885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11335.282692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11335.288562] Lustre: Skipped 3 previous similar messages [11340.748784] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11351.722080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11357.349266] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11360.298888] Lustre: 279335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11367.382836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11375.469759] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11380.346745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11386.404469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11392.051202] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11394.869818] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11403.722993] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11407.948328] Lustre: 281212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11413.392250] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11413.639779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11424.820367] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 01:17:21 (1788844641) [11443.175083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11443.199023] Lustre: Skipped 4 previous similar messages [11443.199482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11443.210796] Lustre: Skipped 3 previous similar messages [11447.738173] Lustre: server umount lustre-MDT0000 complete [11448.297983] LustreError: 278193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11448.338296] LustreError: 278193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [11451.710449] LustreError: 278175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844671 with bad export cookie 8202569861141886910 [11451.717886] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11451.731147] LustreError: 278175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11466.208161] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11466.807988] Lustre: server umount lustre-MDT0001 complete [11486.176836] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11486.424655] Lustre: server umount lustre-OST0000 complete [11490.859804] Lustre: server umount lustre-OST0001 complete [11498.530818] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [11509.871213] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [11560.671568] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [11571.664244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11571.935282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11571.971400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11572.082107] Lustre: lustre-MDT0000: new disk, initializing [11572.215675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11577.145905] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11588.743569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11588.893670] Lustre: 285458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11589.091678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11593.746929] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11606.256402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11606.561579] Lustre: 287092:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11606.661567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11606.668144] Lustre: Skipped 5 previous similar messages [11608.283320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11614.095722] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11626.478214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11626.673751] Lustre: 287963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11628.820081] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11628.831829] Lustre: Skipped 1 previous similar message [11628.959855] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11633.834752] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11644.629260] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11648.902788] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11648.905599] Lustre: Skipped 1 previous similar message [11665.170039] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 01:21:22 (1788844882) [11669.502321] Lustre: server umount lustre-MDT0000 complete [11669.991708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11669.998184] Lustre: Skipped 2 previous similar messages [11672.892407] LustreError: 286264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844892 with bad export cookie 8202569861141894554 [11672.910262] LustreError: 286264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11672.917616] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11673.154695] Lustre: server umount lustre-MDT0001 complete [11687.139092] Lustre: server umount lustre-OST0000 complete [11701.111257] Lustre: server umount lustre-OST0001 complete [11710.829522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11711.380839] LustreError: 290209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11711.408105] LustreError: 290209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11716.215770] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11724.849684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11731.405232] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11739.239304] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11745.502065] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11752.837931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11759.821434] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11767.701573] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11788.456951] loop1: detected capacity change from 0 to 204800000 [11788.740343] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11788.777195] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11788.808067] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11794.884456] loop1: detected capacity change from 0 to 204800000 [11801.635676] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11801.708590] LustreError: 290198:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs6124: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11801.716606] LustreError: 290198:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs6124 [11801.720230] LustreError: 290198:0:(mgs_handler.c:517:mgs_target_reg()) fs6124-MDT0001: Error checking target index: rc = -13 [11801.726639] LustreError: fs6124-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11801.731671] LustreError: 293928:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11801.736616] LustreError: 293928:0:(tgt_mount.c:2132:server_put_super()) no obd fs6124-MDT0001 [11801.740634] LustreError: 293928:0:(tgt_mount.c:116:server_deregister_mount()) fs6124-MDT0001 not registered [11801.767846] Lustre: server umount fs6124-MDT0001 complete [11801.769972] LustreError: 293928:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11811.233123] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11811.330646] Lustre: Setting parameter fs6124-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs6124-MDT0001 [11811.371477] Lustre: srv-fs6124-MDT0001: No data found on store. Initialize space. [11811.375451] Lustre: Skipped 4 previous similar messages [11811.446588] Lustre: fs6124-MDT0001: new disk, initializing [11811.450071] Lustre: Skipped 3 previous similar messages [11811.569654] Lustre: srv-fs6124-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11811.578821] Lustre: 294395:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6124-MDT0001-mdtlov: can't create new llog: rc = -115 [11811.597167] LustreError: 294395:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6124-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11816.225423] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11823.418803] Lustre: server umount fs6124-MDT0001 complete [11837.432652] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11837.938713] loop1: detected capacity change from 0 to 204800000 [11843.384646] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11843.823251] Lustre: 295965:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6124-MDT0001-mdtlov: can't create new llog: rc = -115 [11843.846134] LustreError: 295965:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6124-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11848.224359] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11852.791796] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11853.589899] Lustre: server umount fs6124-MDT0001 complete [11863.532408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11863.547987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11863.563121] Lustre: Skipped 6 previous similar messages [11869.841295] Lustre: server umount lustre-MDT0000 complete [11880.254932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11880.465360] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11885.959953] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11886.073127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11894.834601] loop1: detected capacity change from 0 to 204800000 [11901.138731] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11901.516880] Lustre: 298412:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6124-MDT0001-mdtlov: can't create new llog: rc = -115 [11901.537265] LustreError: 298412:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6124-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11908.501223] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [11914.827166] Lustre: server umount fs6124-MDT0001 complete [11944.929965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11944.935751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11944.947740] Lustre: Skipped 6 previous similar messages [11949.689464] Lustre: server umount lustre-OST0000 complete [11957.737757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11957.748654] LustreError: Skipped 1 previous similar message [11960.045917] Lustre: server umount lustre-MDT0000 complete [11963.882683] LustreError: 290190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845183 with bad export cookie 8202569861141897235 [11963.893342] LustreError: 290190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11963.893945] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11964.286625] Lustre: server umount lustre-MDT0001 complete [11976.170266] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [11979.190976] Key type lgssc unregistered [11979.538273] LNet: 300756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11979.547951] LNetError: 300756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11979.569907] LNet: Removed LNI 192.168.201.143@tcp [11980.345149] Key type .llcrypt unregistered [11980.347896] Key type ._llcrypt unregistered [12001.791892] Key type ._llcrypt registered [12001.794921] Key type .llcrypt registered [12001.975085] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [12003.684505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12003.726522] alg: No test for adler32 (adler32-zlib) [12004.772640] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [12005.029473] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [12006.736198] Key type lgssc registered [12007.907911] Lustre: Echo OBD driver; http://www.lustre.org/ [12015.917569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12015.940708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12017.446859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12021.865856] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12027.875173] LustreError: 302026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12027.896472] LustreError: 302026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12031.283426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12031.920284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12037.123328] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12047.272897] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12055.271273] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12062.595184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12063.046213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12069.596932] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12076.344410] LustreError: 303857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12077.808932] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12081.645091] LustreError: 303857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12086.761102] LustreError: 303858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12092.904754] LustreError: 304530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12092.943198] LustreError: 304530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12103.141307] LustreError: 304563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12103.165399] LustreError: 304563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12107.196303] loop1: detected capacity change from 0 to 204800000 [12107.424261] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12107.441643] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12107.450413] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12113.328706] loop1: detected capacity change from 0 to 204800000 [12119.084769] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12119.295204] Lustre: fs7076-OST0000: new disk, initializing [12119.303624] Lustre: srv-fs7076-OST0000: No data found on store. Initialize space. [12119.310658] Lustre: 305784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12119.409681] Lustre: fs7076-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12119.879109] LustreError: 303855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12119.900809] LustreError: 303855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [12126.810978] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12135.768939] Lustre: server umount fs7076-OST0000 complete [12150.128927] loop1: detected capacity change from 0 to 204800000 [12154.352295] LustreError: 304530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12154.356210] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12154.367748] LustreError: 304530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [12156.703378] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12156.982715] Lustre: fs7076-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12164.649526] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12165.074089] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12174.596119] loop2: detected capacity change from 0 to 204800000 [12174.761700] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12174.778664] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12174.798458] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12180.381539] loop2: detected capacity change from 0 to 204800000 [12186.817965] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12186.922307] LustreError: fs7076-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12186.932728] LustreError: fs7076-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12186.942262] LustreError: 308826:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [12186.954280] LustreError: 308826:0:(tgt_mount.c:2132:server_put_super()) no obd fs7076-OST0001 [12186.965993] LustreError: 308826:0:(tgt_mount.c:116:server_deregister_mount()) fs7076-OST0001 not registered [12187.007266] Lustre: server umount fs7076-OST0001 complete [12187.011486] LustreError: 308826:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12198.592378] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12198.918476] Lustre: fs7076-OST0001: new disk, initializing [12198.929935] Lustre: srv-fs7076-OST0001: No data found on store. Initialize space. [12198.937724] Lustre: 309377:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12199.077484] Lustre: fs7076-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12208.171873] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12217.757332] Lustre: server umount fs7076-OST0000 complete [12220.266793] LustreError: 304563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12220.289034] LustreError: 304563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [12227.600938] Lustre: server umount fs7076-OST0001 complete [12240.688261] loop1: detected capacity change from 0 to 204800000 [12246.656437] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12246.852387] Lustre: fs7076-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12253.905431] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12260.827102] loop2: detected capacity change from 0 to 204800000 [12266.687158] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12273.137802] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12283.259723] Lustre: server umount fs7076-OST0000 complete [12293.537673] Lustre: server umount fs7076-OST0001 complete [12305.902388] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12307.938912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12307.949375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12307.964884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12311.011841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12311.021938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12313.779974] Lustre: server umount lustre-OST0000 complete [12318.700687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12318.709809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12318.732160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12318.738579] Lustre: Skipped 1 previous similar message [12321.253437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12321.254101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12321.278868] Lustre: Skipped 1 previous similar message [12323.983959] Lustre: server umount lustre-MDT0000 complete [12328.637566] LustreError: 310626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845548 with bad export cookie 8280340069137189980 [12328.657173] LustreError: 310626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12328.658745] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12328.964253] Lustre: server umount lustre-MDT0001 complete [12344.148824] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [12348.191311] Key type lgssc unregistered [12348.740464] LNet: 315300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12348.762754] LNetError: 315300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12348.811747] LNet: Removed LNI 192.168.201.143@tcp [12350.465000] Key type .llcrypt unregistered [12350.466663] Key type ._llcrypt unregistered [12385.020989] Key type ._llcrypt registered [12385.027113] Key type .llcrypt registered [12385.140302] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [12405.593169] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [12407.094199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12407.196146] alg: No test for adler32 (adler32-zlib) [12408.336567] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [12408.596251] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [12410.288309] Key type lgssc registered [12411.675899] Lustre: Echo OBD driver; http://www.lustre.org/ [12454.651269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12454.699383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12456.022718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12456.067765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12456.229252] Lustre: lustre-MDT0000: new disk, initializing [12456.326469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12456.340539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12461.680845] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12475.220710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12475.361365] Lustre: 319161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12475.435941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12475.443607] Lustre: Skipped 1 previous similar message [12475.573497] Lustre: lustre-MDT0001: new disk, initializing [12475.728124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12475.784691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12475.820182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12481.092595] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12492.019920] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12498.249464] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12505.169801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12505.524096] Lustre: lustre-OST0000: new disk, initializing [12505.530614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12505.539743] Lustre: 321132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12505.663495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12512.905376] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12514.374231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12514.383605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12514.558987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12523.461926] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12526.104343] LustreError: 317334:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0001_UUID failed: -22 [12526.155423] Lustre: Mounted lustre-client - version 2.17.57_106_g45f4cc0 [12527.218153] Lustre: Unmounted lustre-client [12540.393930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12540.403062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12540.409683] LustreError: Skipped 1 previous similar message [12540.412903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12540.462034] Lustre: Skipped 1 previous similar message [12541.420133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12541.436454] Lustre: Skipped 2 previous similar messages [12543.541066] Lustre: server umount lustre-OST0000 complete [12551.658419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12551.672901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12551.677411] Lustre: Skipped 1 previous similar message [12556.772851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12556.791162] Lustre: Skipped 2 previous similar messages [12561.892937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12561.906489] Lustre: Skipped 1 previous similar message [12562.400304] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12562.636837] Lustre: server umount lustre-MDT0000 complete [12567.443743] LustreError: 319155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845787 with bad export cookie 12153847971520357456 [12567.451094] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12567.463698] LustreError: 319155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12568.150045] Lustre: server umount lustre-MDT0001 complete [12577.569028] LNet: 1 local NIs in recovery (showing 1): 192.168.201.143@tcp [12581.727787] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [12584.798969] Key type lgssc unregistered [12585.105074] LNet: 323296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12585.109838] LNetError: 323296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12657.836749] LNet: Removed LNI 192.168.201.143@tcp [12658.691732] Key type .llcrypt unregistered [12658.693713] Key type ._llcrypt unregistered [12680.323037] Key type ._llcrypt registered [12680.326088] Key type .llcrypt registered [12680.439430] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [12698.082453] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [12699.272556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12699.377219] alg: No test for adler32 (adler32-zlib) [12700.490992] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [12700.722190] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [12702.360225] Key type lgssc registered [12703.475586] Lustre: Echo OBD driver; http://www.lustre.org/ [12739.247172] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:39:15 (1788845955) [12746.018773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12746.053759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12747.398361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12747.442579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12747.595558] Lustre: lustre-MDT0000: new disk, initializing [12747.652545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12747.667590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12752.528729] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12764.085075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12764.285275] Lustre: 326514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12764.362309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12764.365507] Lustre: Skipped 1 previous similar message [12764.466208] Lustre: lustre-MDT0001: new disk, initializing [12764.671130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12764.752077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12764.759067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12770.108078] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12780.014829] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12786.566186] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12794.947424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12795.207900] Lustre: lustre-OST0000: new disk, initializing [12795.219604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12795.228193] Lustre: 328488:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12795.329121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12802.572830] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12804.712476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12804.725626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12804.877326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12813.667480] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12830.698709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12830.699569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12830.727194] LustreError: Skipped 1 previous similar message [12830.740544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12835.667065] Lustre: server umount lustre-OST0000 complete [12840.932375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12840.954265] Lustre: Skipped 2 previous similar messages [12840.956628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12840.976602] Lustre: Skipped 2 previous similar messages [12846.169313] Lustre: server umount lustre-MDT0000 complete [12851.023885] LustreError: 326508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846070 with bad export cookie 4522264998774632934 [12851.032899] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12851.044685] LustreError: 326508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12851.788819] Lustre: server umount lustre-MDT0001 complete [12864.568862] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [12868.080955] Key type lgssc unregistered [12868.427992] LNet: 330393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12868.442235] LNetError: 330393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12868.479740] LNet: Removed LNI 192.168.201.143@tcp [12869.608645] Key type .llcrypt unregistered [12869.610781] Key type ._llcrypt unregistered [12896.828813] Key type ._llcrypt registered [12896.833721] Key type .llcrypt registered [12896.991452] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [12898.531621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12898.588944] alg: No test for adler32 (adler32-zlib) [12899.673551] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [12899.945583] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [12901.629071] Key type lgssc registered [12902.887490] Lustre: Echo OBD driver; http://www.lustre.org/ [12910.386383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12910.399680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12911.875463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12916.569185] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12921.317274] LustreError: 331666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12921.336282] LustreError: 331666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12926.436519] LustreError: 331665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12926.490613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12927.025101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12932.237504] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12940.872098] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12947.116768] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12954.534579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12954.957537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12961.687818] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [12970.566444] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12982.282751] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12984.024708] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12988.394166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12988.412692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12988.431103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12991.972343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12991.976514] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12991.982650] Lustre: Skipped 2 previous similar messages [12992.003294] Lustre: Skipped 1 previous similar message [12993.859870] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.43@tcp (stopping) [12993.863568] Lustre: Skipped 1 previous similar message [12997.094501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12997.107625] Lustre: Skipped 2 previous similar messages [13002.208149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13002.213866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13002.227214] Lustre: Skipped 3 previous similar messages [13002.477097] Lustre: server umount lustre-MDT0000 complete [13004.102778] LustreError: 331660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13006.920993] LustreError: 332402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846226 with bad export cookie 8643856100810747439 [13006.928289] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13006.940198] LustreError: 332402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13007.329112] LustreError: 334166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13007.330428] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13011.365538] LustreError: 334166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13011.369563] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13011.378328] LustreError: 334166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13011.399910] Lustre: Skipped 2 previous similar messages [13013.451161] Lustre: server umount lustre-MDT0001 complete [13018.752549] Lustre: server umount lustre-OST0000 complete [13029.094273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13035.041710] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [13038.852548] Lustre: 335848:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.143@tcp [13041.945362] Lustre: server umount MGS complete [13054.585946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13055.453538] LustreError: 336421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13055.616828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13060.805561] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [13070.215326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13070.580448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13075.759326] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [13086.924722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13087.524141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13094.868832] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [13104.182485] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13106.536880] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13114.695276] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13117.773671] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13126.625437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13126.628140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13126.633858] LustreError: Skipped 1 previous similar message [13126.644310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13126.651974] Lustre: Skipped 1 previous similar message [13126.670098] Lustre: Skipped 1 previous similar message [13130.656423] Lustre: server umount lustre-OST0000 complete [13136.869467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13136.892125] Lustre: Skipped 1 previous similar message [13141.335942] Lustre: server umount lustre-MDT0000 complete [13141.987883] LustreError: 336422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13142.013695] LustreError: 336422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13145.876303] LustreError: 336403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846365 with bad export cookie 8643856100810748279 [13145.882128] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13145.895958] LustreError: 336403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13146.438456] Lustre: server umount lustre-MDT0001 complete [13161.947495] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [13164.876432] Key type lgssc unregistered [13165.181914] LNet: 339860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13165.187543] LNetError: 339860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13165.201704] LNet: Removed LNI 192.168.201.143@tcp [13166.057117] Key type .llcrypt unregistered [13166.059473] Key type ._llcrypt unregistered [13189.685437] Key type ._llcrypt registered [13189.690488] Key type .llcrypt registered [13189.979049] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [13210.834483] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [13212.443433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13212.495069] alg: No test for adler32 (adler32-zlib) [13213.544741] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [13213.839055] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [13215.536338] Key type lgssc registered [13217.266381] Lustre: Echo OBD driver; http://www.lustre.org/ [13259.403486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13259.477677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13260.733154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13260.760594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13260.842920] Lustre: lustre-MDT0000: new disk, initializing [13260.918539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13260.934268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13265.525983] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [13277.849238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13278.017424] Lustre: 343023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13278.070406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13278.074080] Lustre: Skipped 1 previous similar message [13278.201296] Lustre: lustre-MDT0001: new disk, initializing [13278.335688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13278.417446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13278.446081] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13283.651698] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [13295.706993] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13303.550663] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13312.289278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13312.642131] Lustre: lustre-OST0000: new disk, initializing [13312.646195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13312.654836] Lustre: 344992:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13312.741720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13320.737239] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [13322.912977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13322.924512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13323.105815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13331.162891] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13336.002564] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13336.225749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13341.732812] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13341.933592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13348.837191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13348.851280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13348.856221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13348.888938] Lustre: Skipped 1 previous similar message [13350.778955] Lustre: server umount lustre-OST0000 complete [13359.074586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13359.079929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13359.096269] Lustre: Skipped 1 previous similar message [13359.117118] Lustre: Skipped 1 previous similar message [13361.423225] Lustre: server umount lustre-MDT0000 complete [13366.071438] LustreError: 343016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846585 with bad export cookie 10029545470856061512 [13366.073730] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13366.077632] LustreError: 343016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13366.510980] Lustre: server umount lustre-MDT0001 complete [13376.987222] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:49:53 (1788846593) [13398.389832] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [13413.972475] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [13460.428347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13460.818935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13460.872386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13461.023937] Lustre: lustre-MDT0000: new disk, initializing [13461.127489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13461.144147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13466.528433] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [13477.734644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13477.921538] Lustre: 349777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13477.965722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13477.969503] Lustre: Skipped 1 previous similar message [13478.080344] Lustre: lustre-MDT0001: new disk, initializing [13478.226537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13478.297894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13478.318294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13483.778292] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [13494.726402] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13501.709601] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13510.777951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13511.124733] Lustre: lustre-OST0000: new disk, initializing [13511.129462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13511.140646] Lustre: 351746:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13511.267164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13518.938740] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [13520.473791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13520.485485] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13520.637604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13530.512509] Lustre: DEBUG MARKER: oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13572.071194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13572.074989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13572.085667] LustreError: Skipped 1 previous similar message [13572.089288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13572.093376] Lustre: Skipped 1 previous similar message [13572.100930] Lustre: Skipped 2 previous similar messages [13577.203110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13577.209547] Lustre: Skipped 1 previous similar message [13578.121780] Lustre: server umount lustre-OST0000 complete [13585.890471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13585.900306] LustreError: Skipped 1 previous similar message [13585.905139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13585.923582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13589.776299] Lustre: server umount lustre-MDT0000 complete [13592.558782] LustreError: 350951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13592.575693] LustreError: 350951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13595.023158] LustreError: 349768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846814 with bad export cookie 10029545470856062394 [13595.038969] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13595.040788] LustreError: 349768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13595.520208] Lustre: server umount lustre-MDT0001 complete [13611.178966] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [13616.408783] Key type lgssc unregistered [13617.052840] LNet: 353924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13617.074620] LNetError: 353924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13617.105336] LNet: Removed LNI 192.168.201.143@tcp [13618.620341] Key type .llcrypt unregistered [13618.628951] Key type ._llcrypt unregistered [13724.496309] Key type ._llcrypt registered [13724.500417] Key type .llcrypt registered [13724.643481] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [13725.026467] Key type .llcrypt unregistered [13725.030047] Key type ._llcrypt unregistered [13809.435237] Key type ._llcrypt registered [13809.439981] Key type .llcrypt registered [13809.534867] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [13809.789462] Key type .llcrypt unregistered [13809.792254] Key type ._llcrypt unregistered [13851.194409] Key type ._llcrypt registered [13851.201826] Key type .llcrypt registered [13851.343284] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [13851.609319] Key type .llcrypt unregistered [13851.610906] Key type ._llcrypt unregistered [13885.154542] Key type ._llcrypt registered [13885.157291] Key type .llcrypt registered [13885.322615] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [13885.634581] Key type .llcrypt unregistered [13885.640082] Key type ._llcrypt unregistered [13964.950743] Key type ._llcrypt registered [13964.957886] Key type .llcrypt registered [13965.048464] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [13965.378333] Key type .llcrypt unregistered [13965.382362] Key type ._llcrypt unregistered [13999.009683] Key type ._llcrypt registered [13999.015255] Key type .llcrypt registered [13999.151928] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [13999.593608] Key type .llcrypt unregistered [13999.595270] Key type ._llcrypt unregistered [14041.727267] Key type ._llcrypt registered [14041.729601] Key type .llcrypt registered [14041.813757] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [14042.261560] Key type .llcrypt unregistered [14042.263989] Key type ._llcrypt unregistered [14077.924682] Key type ._llcrypt registered [14077.926731] Key type .llcrypt registered [14078.078505] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [14078.296372] Key type .llcrypt unregistered [14078.298162] Key type ._llcrypt unregistered [14112.140762] Key type ._llcrypt registered [14112.142731] Key type .llcrypt registered [14112.248203] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [14112.552659] Key type .llcrypt unregistered [14112.555349] Key type ._llcrypt unregistered [14144.847905] Key type ._llcrypt registered [14144.853352] Key type .llcrypt registered [14144.986265] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [14145.299461] Key type .llcrypt unregistered [14145.302510] Key type ._llcrypt unregistered [14211.596580] Key type ._llcrypt registered [14211.598613] Key type .llcrypt registered [14211.697556] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [14211.926906] Key type .llcrypt unregistered [14211.930691] Key type ._llcrypt unregistered [14247.252668] Key type ._llcrypt registered [14247.255956] Key type .llcrypt registered [14247.382778] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [14247.649526] Key type .llcrypt unregistered [14247.652982] Key type ._llcrypt unregistered [14282.536568] Key type ._llcrypt registered [14282.538280] Key type .llcrypt registered [14282.650778] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [14282.928529] Key type .llcrypt unregistered [14282.930370] Key type ._llcrypt unregistered [14353.737749] Key type ._llcrypt registered [14353.740722] Key type .llcrypt registered [14353.866470] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [14370.592469] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [14371.935418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14371.973054] alg: No test for adler32 (adler32-zlib) [14373.060186] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [14373.293643] LNet: Added LNI 192.168.201.143@tcp [8/256/0/180] [14375.028076] Key type lgssc registered [14376.724443] Lustre: Echo OBD driver; http://www.lustre.org/ [14417.203689] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14417.226189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14418.609538] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14418.649562] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14418.715485] Lustre: tst250fs-MDT0000: new disk, initializing [14418.780228] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14418.798495] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14423.855241] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [14436.502215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14436.673571] Lustre: 374237:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14436.803693] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14436.813029] Lustre: Skipped 1 previous similar message [14436.980438] Lustre: tst250fs-MDT0001: new disk, initializing [14437.136030] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14437.209046] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14437.223152] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14443.849587] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [14449.580092] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14458.452196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14458.896544] Lustre: tst250fs-OST0000: new disk, initializing [14458.900126] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14458.906779] Lustre: 375869:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14458.970140] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14462.177927] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14462.207695] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14462.384439] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14466.492196] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [14479.422266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14479.648891] Lustre: tst250fs-OST0001: new disk, initializing [14479.653192] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14479.662623] Lustre: 376741:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14479.784628] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14486.958367] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_default_debug -1 all [14488.674327] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14488.684292] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14488.797199] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14493.380502] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14495.473519] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14495.481820] Lustre: Skipped 1 previous similar message [14519.792322] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14519.821992] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14519.835800] Lustre: Skipped 2 previous similar messages [14524.386792] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14524.390438] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14524.413966] Lustre: Skipped 2 previous similar messages [14524.417438] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14526.231687] Lustre: server umount tst250fs-MDT0000 complete [14530.029199] LustreError: 374243:0:(ldlm_lib.c:1190:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14530.052095] LustreError: 374243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [14530.312205] LustreError: 374227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847750 with bad export cookie 18288124002570026010 [14530.326569] LustreError: MGC192.168.201.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14530.331878] LustreError: 374227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14530.779222] Lustre: server umount tst250fs-MDT0001 complete [14546.116953] Lustre: server umount tst250fs-OST0000 complete [14561.641655] Lustre: server umount tst250fs-OST0001 complete [14583.512537] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [14593.305706] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [14631.024735] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:10:47 (1788847847) [14633.702792] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14658.170388] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing set_hostid [14672.905634] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing load_modules_local [14716.819607] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14383 sec ========= 02:12:13 (1788847933) [14718.993762] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:12:15 (1788847935) === [14723.676052] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:12:20 (1788847940) === [14775.797613] Lustre: DEBUG MARKER: oleg143-server.virtnet: executing unload_modules_local [14779.134840] Key type lgssc unregistered [14779.493459] LNet: 385912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14779.502254] LNetError: 385912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14779.518277] LNet: Removed LNI 192.168.201.143@tcp [14780.623349] Key type .llcrypt unregistered [14780.625739] Key type ._llcrypt unregistered