[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478713313 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.005000] x2apic enabled [ 0.005011] Switched APIC routing to physical x2apic. [ 0.006017] kvm-guest: setup PV IPIs [ 0.011524] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.012000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.012023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.014011] pid_max: default: 32768 minimum: 301 [ 0.015137] LSM: Security Framework initializing [ 0.016065] Yama: becoming mindful. [ 0.017000] SELinux: Initializing. [ 0.017065] *** VALIDATE selinux *** [ 0.026159] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032665] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034153] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035158] *** VALIDATE tmpfs *** [ 0.038074] *** VALIDATE proc *** [ 0.040204] *** VALIDATE cgroup *** [ 0.041015] *** VALIDATE cgroup2 *** [ 0.043250] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.044300] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.045017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.047034] Spectre V2 : User space: Vulnerable [ 0.048020] Speculative Store Bypass: Vulnerable [ 0.055000] debug: unmapping init [mem 0xffffffffb3a59000-0xffffffffb3a60fff] [ 0.058347] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.060428] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.061033] ... version: 2 [ 0.062026] ... bit width: 48 [ 0.063087] ... generic registers: 4 [ 0.064028] ... value mask: 0000ffffffffffff [ 0.065029] ... max period: 00007fffffffffff [ 0.066027] ... fixed-purpose events: 3 [ 0.067027] ... event mask: 000000070000000f [ 0.068708] rcu: Hierarchical SRCU implementation. [ 0.071820] smp: Bringing up secondary CPUs ... [ 0.073396] x86: Booting SMP configuration: [ 0.074039] .... node #0, CPUs: #1 #2 #3 [ 0.089135] smp: Brought up 1 node, 4 CPUs [ 0.091026] smpboot: Max logical packages: 1 [ 0.092026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147046] node 0 deferred pages initialised in 49ms [ 0.154399] devtmpfs: initialized [ 0.155415] x86/mm: Memory block size: 128MB [ 0.159156] gcov: version magic: 0x41383552 [ 0.162763] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.172104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.173376] pinctrl core: initialized pinctrl subsystem [ 0.174481] [ 0.175014] ************************************************************* [ 0.176023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177018] ** ** [ 0.178029] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.179026] ** ** [ 0.180021] ** This means that this kernel is built to expose internal ** [ 0.181061] ** IOMMU data structures, which may compromise security on ** [ 0.182024] ** your system. ** [ 0.183024] ** ** [ 0.184018] ** If you see this message and you are not debugging the ** [ 0.185017] ** kernel, report this immediately to your vendor! ** [ 0.186027] ** ** [ 0.187018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188025] ************************************************************* [ 0.189783] NET: Registered protocol family 16 [ 0.193905] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195232] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.204149] cpuidle: using governor menu [ 0.207193] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212049] PCI: Using configuration type 1 for base access [ 0.213215] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220218] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228041] cryptd: max_cpu_qlen set to 1000 [ 0.231359] ACPI: Added _OSI(Module Device) [ 0.232147] ACPI: Added _OSI(Processor Device) [ 0.234025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.243574] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.254468] ACPI: Interpreter enabled [ 0.257262] ACPI: PM: (supports S0 S3 S4 S5) [ 0.259022] ACPI: Using IOAPIC for interrupt routing [ 0.264206] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270668] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.285983] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.289111] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.294032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.299194] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.308081] acpiphp: Slot [2] registered [ 0.309322] acpiphp: Slot [5] registered [ 0.312399] acpiphp: Slot [6] registered [ 0.318733] acpiphp: Slot [7] registered [ 0.321624] acpiphp: Slot [8] registered [ 0.326747] acpiphp: Slot [9] registered [ 0.332227] acpiphp: Slot [10] registered [ 0.335231] acpiphp: Slot [3] registered [ 0.336138] acpiphp: Slot [4] registered [ 0.341264] acpiphp: Slot [11] registered [ 0.343464] acpiphp: Slot [12] registered [ 0.345260] acpiphp: Slot [13] registered [ 0.351202] acpiphp: Slot [14] registered [ 0.355014] acpiphp: Slot [15] registered [ 0.358181] acpiphp: Slot [16] registered [ 0.361444] acpiphp: Slot [17] registered [ 0.363220] acpiphp: Slot [18] registered [ 0.364000] acpiphp: Slot [19] registered [ 0.365367] acpiphp: Slot [20] registered [ 0.368583] acpiphp: Slot [21] registered [ 0.370164] acpiphp: Slot [22] registered [ 0.376547] acpiphp: Slot [23] registered [ 0.381211] acpiphp: Slot [24] registered [ 0.383166] acpiphp: Slot [25] registered [ 0.386245] acpiphp: Slot [26] registered [ 0.391192] acpiphp: Slot [27] registered [ 0.394183] acpiphp: Slot [28] registered [ 0.395000] acpiphp: Slot [29] registered [ 0.397197] acpiphp: Slot [30] registered [ 0.399818] acpiphp: Slot [31] registered [ 0.404568] PCI host bridge to bus 0000:00 [ 0.408035] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.411078] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.415037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.416029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.421095] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.429057] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.431408] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.439151] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.451226] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.475023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.489082] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.496298] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.505032] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.512031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.518640] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.523384] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.530054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.534707] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.553023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.576022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.587020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.596818] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.604132] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.613023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.645027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.665000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.682024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.697032] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.728024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.748000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.765022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.776023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.807023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.843223] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.855017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.869017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.896018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.916115] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.950019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.967021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.009020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.033000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.050016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.071018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.088017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.095000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.104898] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.109636] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.112795] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.115243] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.125083] iommu: Default domain type: Passthrough [ 1.127514] SCSI subsystem initialized [ 1.130366] ACPI: bus type USB registered [ 1.131213] usbcore: registered new interface driver usbfs [ 1.133095] usbcore: registered new interface driver hub [ 1.134091] usbcore: registered new device driver usb [ 1.137330] pps_core: LinuxPPS API ver. 1 registered [ 1.138016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.143064] PTP clock support registered [ 1.147156] EDAC MC: Ver: 3.0.0 [ 1.149166] PCI: Using ACPI for IRQ routing [ 1.151068] NetLabel: Initializing [ 1.152010] NetLabel: domain hash size = 128 [ 1.156015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.158078] NetLabel: unlabeled traffic allowed by default [ 1.169127] vgaarb: loaded [ 1.176344] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.178014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.186008] clocksource: Switched to clocksource kvm-clock [ 1.448537] VFS: Disk quotas dquot_6.6.0 [ 1.450875] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.464865] *** VALIDATE ramfs *** [ 1.468957] *** VALIDATE hugetlbfs *** [ 1.474561] pnp: PnP ACPI init [ 1.489986] pnp: PnP ACPI: found 6 devices [ 1.531350] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.540422] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.548544] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.555611] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.564073] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.568626] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.572873] NET: Registered protocol family 2 [ 1.578530] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.587901] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.598107] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.610728] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.615634] TCP: Hash tables configured (established 65536 bind 65536) [ 1.623043] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.629126] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.634686] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.641119] NET: Registered protocol family 1 [ 1.647180] RPC: Registered named UNIX socket transport module. [ 1.652123] RPC: Registered udp transport module. [ 1.655066] RPC: Registered tcp transport module. [ 1.657513] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.662393] NET: Registered protocol family 44 [ 1.665073] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.668681] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.673615] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.679649] PCI: CLS 0 bytes, default 64 [ 1.682364] Unpacking initramfs... [ 4.005022] hrtimer: interrupt took 3490367 ns [ 4.888813] debug: unmapping init [mem 0xffff931a3cc54000-0xffff931a3ffbffff] [ 4.897398] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.906704] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.917561] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.570590] Initialise system trusted keyrings [ 7.576047] Key type blacklist registered [ 7.581688] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.603607] zbud: loaded [ 7.613225] *** VALIDATE nfs *** [ 7.614397] *** VALIDATE nfs4 *** [ 7.620577] pstore: using deflate compression [ 7.634154] Platform Keyring initialized [ 8.085717] NET: Registered protocol family 38 [ 8.093389] Key type asymmetric registered [ 8.100103] Asymmetric key parser 'x509' registered [ 8.107866] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 8.122987] io scheduler mq-deadline registered [ 8.130968] io scheduler kyber registered [ 8.135062] io scheduler bfq registered [ 8.142241] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 8.145896] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 8.170663] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 8.185343] ACPI: Power Button [PWRF] [ 8.208292] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 8.241345] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 8.262564] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 8.282162] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 8.320059] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.383718] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.432103] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.443752] Non-volatile memory driver v1.3 [ 8.447307] Linux agpgart interface v0.103 [ 8.538819] virtio_blk virtio1: [vda] 139528 512-byte logical blocks (71.4 MB/68.1 MiB) [ 8.544936] vda: detected capacity change from 0 to 71438336 [ 8.589976] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.598697] vdb: detected capacity change from 0 to 1073741824 [ 8.674220] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.678955] vdc: detected capacity change from 0 to 2621440000 [ 8.737487] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.746316] vdd: detected capacity change from 0 to 2621440000 [ 8.780162] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.784032] vde: detected capacity change from 0 to 4294967296 [ 8.817952] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.824229] vdf: detected capacity change from 0 to 4294967296 [ 8.843037] libphy: Fixed MDIO Bus: probed [ 8.854101] usbcore: registered new interface driver usbserial_generic [ 8.858965] usbserial: USB Serial support registered for generic [ 8.863341] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.869986] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.872667] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.877308] mousedev: PS/2 mouse device common for all mice [ 8.881986] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.886905] rtc_cmos 00:05: RTC can wake from S4 [ 8.891695] rtc_cmos 00:05: registered as rtc0 [ 8.893723] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.897052] intel_pstate: CPU model not supported [ 8.901950] hid: raw HID events driver (C) Jiri Kosina [ 8.905837] usbcore: registered new interface driver usbhid [ 8.907989] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.910461] usbhid: USB HID core driver [ 8.917571] drop_monitor: Initializing network drop monitor service [ 8.921210] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.926690] Initializing XFRM netlink socket [ 8.929090] NET: Registered protocol family 10 [ 8.932798] Segment Routing with IPv6 [ 8.934958] NET: Registered protocol family 17 [ 8.937549] mpls_gso: MPLS GSO support [ 8.944212] RAS: Correctable Errors collector initialized. [ 8.946854] AVX version of gcm_enc/dec engaged. [ 8.948920] AES CTR mode by8 optimization enabled [ 9.045900] sched_clock: Marking stable (9045878943, 0)->(10596224121, -1550345178) [ 9.050946] registered taskstats version 1 [ 9.053488] Loading compiled-in X.509 certificates [ 9.070534] zswap: loaded using pool lzo/zbud [ 9.156991] Key type big_key registered [ 9.205106] Key type encrypted registered [ 9.206839] ima: No TPM chip found, activating TPM-bypass! [ 9.212256] ima: Allocated hash algorithm: sha1 [ 9.215493] ima: No architecture policies found [ 9.219687] evm: Initialising EVM extended attributes: [ 9.224644] evm: security.selinux [ 9.227210] evm: security.ima [ 9.230666] evm: security.capability [ 9.232802] evm: HMAC attrs: 0x1 [ 9.237083] rtc_cmos 00:05: setting system clock to 2026-06-09 09:06:15 UTC (1780995975) [ 9.251885] debug: unmapping init [mem 0xffffffffb4a03000-0xffffffffb4bfffff] [ 9.258687] debug: unmapping init [mem 0xffffffffb3782000-0xffffffffb3a58fff] [ 9.285477] Write protecting the kernel read-only data: 28672k [ 9.295498] debug: unmapping init [mem 0xffffffffb1e03000-0xffffffffb1ffffff] [ 9.300051] debug: unmapping init [mem 0xffffffffb2714000-0xffffffffb27fffff] [ 9.424103] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.460890] systemd[1]: Detected virtualization kvm. [ 9.465566] systemd[1]: Detected architecture x86-64. [ 9.470205] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.548494] systemd[1]: No hostname configured. [ 9.552255] systemd[1]: Set hostname to . [ 9.562054] random: systemd: uninitialized urandom read (16 bytes read) [ 9.569133] systemd[1]: Initializing machine ID from random generator. [ 9.716079] random: ln: uninitialized urandom read (6 bytes read) [ 9.956344] random: systemd: uninitialized urandom read (16 bytes read) [ 9.966574] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 10.020750] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 10.070274] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 12.110583] device-mapper: uevent: version 1.0.3 [ 12.120967] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 14.578011] random: fast init done [ 14.621668] virtio_net virtio0 ens2: renamed from eth0 [ 14.840239] scsi host0: ata_piix [ 14.943329] scsi host1: ata_piix [ 14.944972] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.947915] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.741232] random: crng init done [ 19.744497] random: 7 urandom warning(s) missed due to ratelimiting [ 20.722619] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 22.893271] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.530706] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.500484] SELinux: Disabled at runtime. [ 26.662770] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.692784] systemd[1]: Detected virtualization kvm. [ 26.695223] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.587816] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.598978] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.617152] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.629200] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.640181] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.658897] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.673829] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ 29.435561] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 30.331925] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.477724] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.694105] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.076648] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.194806] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.863302] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 38.870868] NFS: Registering the id_resolver key type [ 38.875749] Key type id_resolver registered [ 38.878481] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg335-server login: [ 97.885534] libcfs: loading out-of-tree module taints kernel. [ 97.931567] Key type ._llcrypt registered [ 97.933654] Key type .llcrypt registered [ 98.030966] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 116.899342] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 118.655242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 118.685649] alg: No test for adler32 (adler32-zlib) [ 120.419566] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 121.755152] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 123.639656] Key type lgssc registered [ 125.842748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 146.087763] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 188.710191] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 201.988489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 202.033534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.355924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 203.435825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 203.542895] Lustre: lustre-MDT0000: new disk, initializing [ 203.686358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.705799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.854929] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.751857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.877718] Lustre: 6360:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.915646] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 220.920561] Lustre: Skipped 1 previous similar message [ 221.056736] Lustre: lustre-MDT0001: new disk, initializing [ 221.165452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.203092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 221.227385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.948330] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.725355] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 240.161373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.376898] Lustre: lustre-OST0000: new disk, initializing [ 240.386329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 240.392165] Lustre: 8264:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.451744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.398350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.405631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 242.527341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 246.587623] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.403247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 264.740248] Lustre: lustre-OST0001: new disk, initializing [ 264.745974] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 264.763290] Lustre: 9318:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 264.928328] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 271.938076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 271.955550] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 272.136138] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 273.023710] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.079584] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.837645] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 302.001410] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing check_logdir /tmp/testlogs/ [ 306.287700] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing yml_node [ 310.586690] Lustre: DEBUG MARKER: Client: 2.17.54.1 [ 313.118147] Lustre: DEBUG MARKER: MDS: 2.17.54.1 [ 315.509117] Lustre: DEBUG MARKER: OSS: 2.17.54.1 [ 317.181167] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 9 05:11:22 EDT 2026 [ 332.197572] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 333.689813] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 354.274146] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.277698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 354.289246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.295323] Lustre: Skipped 2 previous similar messages [ 356.235152] Lustre: server umount lustre-MDT0000 complete [ 359.695988] LustreError: 6353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780996325 with bad export cookie 16212196433529465995 [ 359.702711] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 359.715844] LustreError: 6353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 364.512893] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.514806] LustreError: 8392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.535528] Lustre: Skipped 1 previous similar message [ 364.540339] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 364.540348] Lustre: Skipped 3 previous similar messages [ 364.581402] LustreError: 8392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 369.631734] LustreError: 6369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.637024] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 369.650462] LustreError: 6369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 369.672027] Lustre: Skipped 2 previous similar messages [ 370.658406] LustreError: 6367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.685337] LustreError: 6367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 374.239224] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 374.500253] Lustre: server umount lustre-MDT0001 complete [ 378.508808] Lustre: server umount lustre-OST0000 complete [ 382.253669] Lustre: server umount lustre-OST0001 complete [ 389.051378] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 398.649977] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 431.258364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.517720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 431.554832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 431.668993] Lustre: lustre-MDT0000: new disk, initializing [ 431.819573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.848238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 436.502499] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 447.215988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.351812] Lustre: 14692:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 447.406686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 447.410795] Lustre: Skipped 1 previous similar message [ 447.481421] Lustre: lustre-MDT0001: new disk, initializing [ 447.575379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 447.607446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 447.616295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 451.765427] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 460.266442] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 464.627108] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 473.660051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 474.285204] Lustre: lustre-OST0000: new disk, initializing [ 474.296855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 474.309396] Lustre: 16438:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 474.474416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 475.484322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 475.496991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 475.662333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 481.824748] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 490.486091] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 495.972527] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 496.284588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 501.076441] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 501.268821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 506.336636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 506.342943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.363231] LustreError: Skipped 1 previous similar message [ 506.368853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.368862] Lustre: Skipped 2 previous similar messages [ 506.422499] Lustre: Skipped 2 previous similar messages [ 509.462614] Lustre: server umount lustre-OST0000 complete [ 514.019255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 514.045124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.063397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 514.071108] Lustre: Skipped 1 previous similar message [ 525.795270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 525.799112] Lustre: Skipped 4 previous similar messages [ 528.351242] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 528.607631] Lustre: server umount lustre-MDT0000 complete [ 530.913792] LustreError: 14705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.948181] LustreError: 14705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 532.937733] LustreError: 14684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780996499 with bad export cookie 16212196433529467528 [ 532.940456] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 532.948722] LustreError: 14684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 533.406871] Lustre: server umount lustre-MDT0001 complete [ 540.181443] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 05:15:04 (1780996504) [ 562.247425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.002888] LustreError: 19165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.083894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.594379] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 573.411076] LustreError: 19166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.439157] LustreError: 19166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 576.443746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.518677] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 587.223987] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 592.476235] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 595.451662] Lustre: Failing over lustre-MDT0000 [ 596.294081] Lustre: server umount lustre-MDT0000 complete [ 597.488735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 597.496222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.504544] Lustre: Skipped 1 previous similar message [ 597.507440] LustreError: 19165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.856250] Lustre: 3513:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780996564/real 1780996564] req@ffff931a88c6e680 x1867509753084288/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780996580 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 613.893083] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.521480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.188075] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe0fd4ac2a960f80e [ 624.211875] Lustre: MGC192.168.203.135@tcp: Connection restored to 0@lo (at 0@lo) [ 624.737828] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 629.739384] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 629.748820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 629.766279] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 630.064264] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 638.776613] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 644.071860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.087017] Lustre: Skipped 1 previous similar message [ 644.097702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 644.105161] Lustre: Skipped 2 previous similar messages [ 657.376270] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 657.547894] Lustre: server umount lustre-MDT0000 complete [ 661.478100] LustreError: 19876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 661.508891] LustreError: 19876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 661.553449] LustreError: 19147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780996627 with bad export cookie 16212196433529468942 [ 661.556749] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 661.566766] LustreError: 19147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 662.027942] Lustre: server umount lustre-MDT0001 complete [ 671.696904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.366304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 672.375231] Lustre: Skipped 2 previous similar messages [ 677.848157] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 688.343110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.720375] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 700.700259] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 705.291981] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 707.402679] Lustre: Failing over lustre-MDT0000 [ 707.768796] Lustre: server umount lustre-MDT0000 complete [ 708.579519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.594532] Lustre: Skipped 2 previous similar messages [ 723.944643] Lustre: 3510:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780996674/real 1780996674] req@ffff931ab7971880 x1867509753114624/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780996690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 724.025324] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 729.056441] LustreError: 22164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 729.077655] LustreError: 22164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 729.372657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 734.181194] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe0fd4ac2a960fbb8 [ 734.196201] Lustre: MGC192.168.203.135@tcp: Connection restored to 0@lo (at 0@lo) [ 734.211603] Lustre: Skipped 1 previous similar message [ 734.578419] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 739.706389] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 739.809873] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 739.833925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 739.858173] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 748.801172] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 754.145381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.151287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 754.170241] Lustre: Skipped 1 previous similar message [ 754.188394] Lustre: Skipped 7 previous similar messages [ 759.869813] Lustre: server umount lustre-MDT0000 complete [ 763.678588] LustreError: 22149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780996729 with bad export cookie 16212196433529469880 [ 763.698258] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 764.133359] Lustre: server umount lustre-MDT0001 complete [ 772.333966] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 05:18:57 (1780996737) [ 788.638755] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 801.003291] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 841.642414] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 852.935435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 853.242450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 853.307145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 853.434759] Lustre: lustre-MDT0000: new disk, initializing [ 853.541921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 853.550669] Lustre: Skipped 2 previous similar messages [ 853.571725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 858.074413] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 869.293874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.529703] Lustre: 28152:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 869.941921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 876.423894] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 886.021991] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 891.204280] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 899.250204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 899.731710] Lustre: 29895:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 901.611627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 901.680248] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 901.785190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 901.809293] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 907.418161] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 915.200679] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 920.720997] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 942.562878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.571322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 942.582261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 942.582268] Lustre: Skipped 2 previous similar messages [ 953.823940] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 954.221597] Lustre: server umount lustre-OST0000 complete [ 972.255225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 972.501863] Lustre: server umount lustre-MDT0000 complete [ 973.291542] LustreError: 29981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.329672] LustreError: 29981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 976.927883] LustreError: 28145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780996943 with bad export cookie 16212196433529470293 [ 976.954425] LustreError: 28145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 976.961918] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 977.632898] Lustre: server umount lustre-MDT0001 complete [ 990.888875] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 993.822477] Key type lgssc unregistered [ 994.307595] LNet: 31749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.334299] LNetError: 31749:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.365508] LNet: Removed LNI 192.168.203.135@tcp [ 995.301166] Key type .llcrypt unregistered [ 995.303531] Key type ._llcrypt unregistered [ 1017.523543] Key type ._llcrypt registered [ 1017.526930] Key type .llcrypt registered [ 1017.619240] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 1018.765151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1018.814757] alg: No test for adler32 (adler32-zlib) [ 1019.851381] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 1020.014555] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 1021.671290] Key type lgssc registered [ 1022.778976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1029.459760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1029.473799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.997260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.173933] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1041.379307] LustreError: 33147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.398576] LustreError: 33147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1042.281208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.677145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1047.179493] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1053.581069] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1058.760168] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1066.642783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1067.041964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.090175] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1073.033438] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1076.326429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1079.017215] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1094.344085] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 05:24:18 (1780997058) [ 1096.722903] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1098.546666] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1106.325594] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 05:24:30 (1780997070) [ 1115.775276] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 05:24:40 (1780997080) [ 1116.488441] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1126.135550] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1126.141925] Lustre: Skipped 2 previous similar messages [ 1127.671948] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 05:24:52 (1780997092) [ 1131.789501] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1131.805057] Lustre: Skipped 3 previous similar messages [ 1149.095180] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1149.102889] Lustre: Skipped 6 previous similar messages [ 1151.472349] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 05:25:16 (1780997116) [ 1153.783299] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1156.800348] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1164.551942] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 05:25:29 (1780997129) [ 1166.250891] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1166.254893] Lustre: Skipped 3 previous similar messages [ 1178.678334] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 05:25:43 (1780997143) [ 1193.953162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1193.959496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.963629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1193.970289] Lustre: Skipped 1 previous similar message [ 1197.191032] Lustre: server umount lustre-OST0000 complete [ 1201.647805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1201.660626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.669885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.684493] Lustre: Skipped 1 previous similar message [ 1204.202234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1204.208219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.881377] Lustre: server umount lustre-MDT0000 complete [ 1209.316741] LustreError: 33144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.340406] LustreError: 33144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1210.970380] LustreError: 33128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780997177 with bad export cookie 6691812425966917571 [ 1210.972986] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1210.987199] LustreError: 33128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1211.292369] Lustre: server umount lustre-MDT0001 complete [ 1222.060340] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 1224.512168] Key type lgssc unregistered [ 1224.820172] LNet: 40646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.829848] LNetError: 40646:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.894333] LNet: Removed LNI 192.168.203.135@tcp [ 1226.734484] Key type .llcrypt unregistered [ 1226.737585] Key type ._llcrypt unregistered [ 1244.139550] Key type ._llcrypt registered [ 1244.145588] Key type .llcrypt registered [ 1244.278953] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 1259.272431] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 1260.608239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1260.641095] alg: No test for adler32 (adler32-zlib) [ 1261.796330] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 1262.202577] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 1264.071170] Key type lgssc registered [ 1265.627812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1303.622916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1303.676646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.987392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1305.013599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1305.062946] Lustre: lustre-MDT0000: new disk, initializing [ 1305.133246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1305.217442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1308.712444] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1318.550870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.693368] Lustre: 43765:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1318.788129] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1318.792604] Lustre: Skipped 1 previous similar message [ 1318.906324] Lustre: lustre-MDT0001: new disk, initializing [ 1319.016539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1319.060341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1319.094208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1323.637807] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1330.552500] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1334.143622] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1342.458535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1342.834687] Lustre: lustre-OST0000: new disk, initializing [ 1342.843863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1342.857423] Lustre: 45507:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1343.008545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.118428] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1350.234786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1350.248128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1350.408112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1358.170577] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1362.263301] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1362.453953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1367.161701] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1367.290183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1371.116760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1371.117254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1371.152720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1371.169377] LustreError: Skipped 1 previous similar message [ 1375.756180] Lustre: server umount lustre-OST0000 complete [ 1381.288302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.306295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1381.318381] Lustre: Skipped 2 previous similar messages [ 1381.344849] Lustre: Skipped 2 previous similar messages [ 1386.469027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1386.489052] Lustre: Skipped 1 previous similar message [ 1387.662947] Lustre: server umount lustre-MDT0000 complete [ 1392.007788] LustreError: 43757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780997358 with bad export cookie 10939174954819201553 [ 1392.015841] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1392.022930] LustreError: 43757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1392.584929] Lustre: server umount lustre-MDT0001 complete [ 1402.387902] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 05:29:27 (1780997367) [ 1410.055353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.653150] LustreError: 47820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1410.672651] LustreError: 47820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1410.762778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1415.616576] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1416.164331] LustreError: 47821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.263234] LustreError: 47820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.164736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.378688] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1425.587523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1430.568764] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1437.599394] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1442.641726] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1451.403395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1451.811891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.969824] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1463.649221] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1472.519901] LustreError: 50010:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1473.397510] Lustre: Modifying parameter general.timeout=1 in log params [ 1475.088440] Lustre: Modifying parameter general.timeout=3 in log params [ 1475.094388] Lustre: Skipped 1 previous similar message [ 1477.633400] Lustre: Modifying parameter general.timeout=6 in log params [ 1477.638837] Lustre: Skipped 2 previous similar messages [ 1482.155072] Lustre: Modifying parameter general.timeout=11 in log params [ 1482.157908] Lustre: Skipped 4 previous similar messages [ 1490.829287] Lustre: Modifying parameter general.timeout=20 in log params [ 1490.836983] Lustre: Skipped 8 previous similar messages [ 1507.067279] Lustre: Modifying parameter general.timeout=35 in log params [ 1507.080826] Lustre: Skipped 14 previous similar messages [ 1539.264592] Lustre: Modifying parameter general.timeout=70 in log params [ 1539.271654] Lustre: Skipped 34 previous similar messages [ 1603.640523] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 05:32:48 (1780997568) [ 1604.618431] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1604.638531] Lustre: Skipped 60 previous similar messages [ 1606.370865] Lustre: 56959:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1608.400506] Lustre: 57057:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1610.557694] Lustre: 57155:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1615.855407] Lustre: 57469:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1615.865234] Lustre: 57469:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 2 previous similar messages [ 1641.452591] Lustre: 58830:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1641.468513] LustreError: 58830:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1644.910853] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 05:33:29 (1780997609) [ 1659.874996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.881935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1659.890914] Lustre: Skipped 1 previous similar message [ 1659.902906] Lustre: Skipped 2 previous similar messages [ 1663.943669] Lustre: server umount lustre-MDT0000 complete [ 1664.995542] LustreError: 49594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1665.019838] LustreError: 49594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1667.330368] LustreError: 47802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780997633 with bad export cookie 10939174954819202435 [ 1667.335337] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1667.344438] LustreError: 47802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1667.710920] Lustre: server umount lustre-MDT0001 complete [ 1681.979235] Lustre: server umount lustre-OST0000 complete [ 1709.432722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.897248] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1710.189134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1714.805967] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1723.304213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.377641] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1723.421033] Lustre: 61365:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1728.201917] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1733.369941] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1736.998469] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1743.860120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1744.039682] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1744.344338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.348407] Lustre: Skipped 1 previous similar message [ 1751.216028] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1757.197602] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1758.625620] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1758.636227] Lustre: Skipped 22 previous similar messages [ 1758.687894] Lustre: 63590:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1767.904127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1767.910253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.925102] Lustre: Skipped 1 previous similar message [ 1767.933283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1771.586670] Lustre: server umount lustre-OST0000 complete [ 1775.072367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1775.078401] LustreError: Skipped 1 previous similar message [ 1775.083187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.100860] Lustre: Skipped 1 previous similar message [ 1781.230469] Lustre: server umount lustre-MDT0000 complete [ 1783.779213] LustreError: 61376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1783.814601] LustreError: 61376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1785.022089] LustreError: 61355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780997751 with bad export cookie 10939174954819205543 [ 1785.023186] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1785.039897] LustreError: 61355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1785.434858] Lustre: server umount lustre-MDT0001 complete [ 1795.028254] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 1797.663440] Key type lgssc unregistered [ 1797.901075] LNet: 64783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.912147] LNetError: 64783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.953953] LNet: Removed LNI 192.168.203.135@tcp [ 1799.797535] Key type .llcrypt unregistered [ 1799.800201] Key type ._llcrypt unregistered [ 1825.599811] Key type ._llcrypt registered [ 1825.604578] Key type .llcrypt registered [ 1825.763629] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 1827.402849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1827.460150] alg: No test for adler32 (adler32-zlib) [ 1828.593996] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 1828.941668] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 1830.599393] Key type lgssc registered [ 1831.761765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1839.533571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1839.571492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.208783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.454672] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1851.366652] LustreError: 66180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1851.407060] LustreError: 66180:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1854.304361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.133846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1859.736347] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1866.237569] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1870.162881] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1877.107143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.623932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.571338] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1890.473799] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1899.142167] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1911.265698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1911.275444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.287316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1911.782647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.795185] Lustre: Skipped 1 previous similar message [ 1911.799839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1911.807373] Lustre: Skipped 1 previous similar message [ 1914.356925] Lustre: server umount lustre-MDT0000 complete [ 1916.904621] LustreError: 66177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.943769] LustreError: 66177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1918.252567] LustreError: 66161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780997884 with bad export cookie 10661097528158582797 [ 1918.264464] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1918.266021] LustreError: 66161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1918.826987] Lustre: server umount lustre-MDT0001 complete [ 1933.619153] Lustre: server umount lustre-OST0000 complete [ 1961.005027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.361660] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1961.396282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1961.788157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1966.786708] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1977.077332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.238579] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1977.298752] Lustre: 70713:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1977.963622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1982.566938] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1988.256982] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1992.208529] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2000.327169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.558060] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2001.036567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.323377] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2014.550977] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2016.407744] Lustre: 72948:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2018.353644] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2023.396901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2023.405548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.417849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2023.462259] Lustre: Skipped 1 previous similar message [ 2028.517531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2028.524748] Lustre: Skipped 1 previous similar message [ 2029.628339] Lustre: server umount lustre-OST0000 complete [ 2038.760937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.779104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.782540] Lustre: Skipped 1 previous similar message [ 2038.796536] Lustre: Skipped 2 previous similar messages [ 2040.791368] Lustre: server umount lustre-MDT0000 complete [ 2043.873307] LustreError: 70718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.890297] LustreError: 70718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2045.121907] LustreError: 70705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780998011 with bad export cookie 10661097528158583630 [ 2045.127810] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2045.144153] LustreError: 70705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2045.545637] Lustre: server umount lustre-MDT0001 complete [ 2058.045941] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 2060.889512] Key type lgssc unregistered [ 2061.249399] LNet: 74137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.263596] LNetError: 74137:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.290342] LNet: Removed LNI 192.168.203.135@tcp [ 2062.145355] Key type .llcrypt unregistered [ 2062.151605] Key type ._llcrypt unregistered [ 2087.561494] Key type ._llcrypt registered [ 2087.566847] Key type .llcrypt registered [ 2087.702621] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2089.033359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2089.172781] alg: No test for adler32 (adler32-zlib) [ 2090.320601] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 2090.577206] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 2092.227522] Key type lgssc registered [ 2093.630169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2102.529126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2102.555608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.199379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2109.216845] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2114.531428] LustreError: 75537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.554984] LustreError: 75537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2119.661597] LustreError: 75536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.019896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.457415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2125.381481] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2131.796738] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2135.717649] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2143.403118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2144.331520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.588671] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2157.330707] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2164.083564] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2165.036700] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2166.639308] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2166.642910] Lustre: Skipped 1 previous similar message [ 2169.486056] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2169.489030] Lustre: Skipped 2 previous similar messages [ 2174.153640] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2174.158963] Lustre: Skipped 4 previous similar messages [ 2182.785256] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2182.790241] Lustre: Skipped 8 previous similar messages [ 2199.669852] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2199.677672] Lustre: Skipped 17 previous similar messages [ 2231.829837] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2231.835396] Lustre: Skipped 33 previous similar messages [ 2264.845053] Lustre: 83290:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2268.152158] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:43:52 (1780998232) [ 2269.788746] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2271.702891] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:43:56 (1780998236) [ 2277.344793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2277.353064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.354509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2277.377683] Lustre: Skipped 1 previous similar message [ 2277.399731] Lustre: Skipped 1 previous similar message [ 2282.479440] LustreError: 77614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.509651] LustreError: 77614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2282.642103] Lustre: server umount lustre-OST0000 complete [ 2289.122788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2289.147463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.162798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.724701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.726618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.552610] Lustre: server umount lustre-MDT0000 complete [ 2297.825607] LustreError: 75531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.844948] LustreError: 75531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2298.718972] LustreError: 75516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780998264 with bad export cookie 14131976401749728692 [ 2298.728041] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2298.738365] LustreError: 75516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2299.118897] Lustre: server umount lustre-MDT0001 complete [ 2311.458235] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 2314.764207] Key type lgssc unregistered [ 2315.041186] LNet: 84526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.052498] LNetError: 84526:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2315.075941] LNet: Removed LNI 192.168.203.135@tcp [ 2316.051326] Key type .llcrypt unregistered [ 2316.059792] Key type ._llcrypt unregistered [ 2320.643363] Key type ._llcrypt registered [ 2320.645437] Key type .llcrypt registered [ 2320.756944] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2325.863390] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules [ 2327.284569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2327.323455] alg: No test for adler32 (adler32-zlib) [ 2328.371934] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 2328.685518] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 2330.431539] Key type lgssc registered [ 2331.950662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2333.358062] LustreError: 85705:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2350.726803] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2354.819247] LustreError: 85705:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2361.629642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2361.676208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.322210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2367.820563] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2373.614391] LustreError: 86708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.629285] LustreError: 86708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2377.296885] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:45:42 (1780998342) [ 2378.727692] LustreError: 86707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.469415] LustreError: 86739:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2382.489806] LustreError: 86739:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2382.503367] LustreError: 86739:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2382.997742] Lustre: server umount lustre-MDT0000 complete [ 2397.634620] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 2400.594613] Key type lgssc unregistered [ 2400.920950] LNet: 88198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.926257] LNetError: 88198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.947492] LNet: Removed LNI 192.168.203.135@tcp [ 2401.785971] Key type .llcrypt unregistered [ 2401.792736] Key type ._llcrypt unregistered [ 2421.881940] Key type ._llcrypt registered [ 2421.891649] Key type .llcrypt registered [ 2422.019823] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2423.648606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2423.690747] alg: No test for adler32 (adler32-zlib) [ 2424.940409] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 2425.211963] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 2426.983233] Key type lgssc registered [ 2429.008678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2437.888674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2437.903768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.646816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2444.427220] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2448.883435] LustreError: 89402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.916964] LustreError: 89402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2453.984015] LustreError: 89403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.328601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2455.392922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2461.638732] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2469.377704] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2474.251696] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2481.607036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2482.343612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.730375] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2494.778973] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2552.286539] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:48:37 (1780998517) [ 2554.094707] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2556.239666] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:48:40 (1780998520) [ 2562.528570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2562.534947] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.540944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2562.571504] Lustre: Skipped 2 previous similar messages [ 2567.161186] Lustre: server umount lustre-MDT0000 complete [ 2567.652219] LustreError: 89398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.681178] LustreError: 89398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2571.036570] LustreError: 89382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780998537 with bad export cookie 10088382497738032779 [ 2571.046947] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2571.054894] LustreError: 89382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2571.761773] Lustre: server umount lustre-MDT0001 complete [ 2586.591762] Lustre: server umount lustre-OST0000 complete [ 2599.472554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.006232] LustreError: 93359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.272261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2604.889939] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2605.538908] LustreError: 93360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.629201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2614.529717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2619.722648] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2626.324267] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2630.821555] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2644.257724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2644.496511] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2644.506327] LustreError: 93349:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2644.528372] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2644.535596] LustreError: 95218:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2644.551493] LustreError: 95218:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2644.570237] LustreError: 95218:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2644.638251] Lustre: server umount lustre-OST0000 complete [ 2644.641915] LustreError: 95218:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2653.424792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2653.730378] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2653.743229] LustreError: 93346:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2653.776469] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2653.799805] LustreError: 95685:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2653.810603] LustreError: 95685:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2653.817612] LustreError: 95685:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2653.966160] Lustre: server umount lustre-OST0000 complete [ 2653.976841] LustreError: 95685:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2663.856316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2664.090487] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2664.094364] Lustre: Found index 0 for lustre-OST0000, updating log [ 2664.098962] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2664.182872] Lustre: lustre-OST0000: new disk, initializing [ 2664.186108] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2664.210398] Lustre: 96220:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2664.577594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.500752] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2674.695437] Lustre: Failing over lustre-OST0000 [ 2674.948115] Lustre: server umount lustre-OST0000 complete [ 2680.800657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2680.813295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.832260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2680.843146] Lustre: Skipped 2 previous similar messages [ 2684.386469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2684.391071] Lustre: Skipped 2 previous similar messages [ 2689.520328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2689.531325] Lustre: Skipped 2 previous similar messages [ 2693.087153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2693.321457] Lustre: server umount lustre-MDT0000 complete [ 2694.628139] LustreError: 94073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.646429] LustreError: 94073:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2697.061224] LustreError: 93339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780998663 with bad export cookie 10088382497738035523 [ 2697.066073] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2697.072970] LustreError: 93339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2697.385364] Lustre: server umount lustre-MDT0001 complete [ 2705.369424] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:51:10 (1780998670) [ 2724.076921] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2733.726534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.172802] LustreError: 98531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.303791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.091353] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2747.791710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.601410] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2756.640594] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2764.225730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2765.121064] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2771.646439] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2773.285434] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2773.298655] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2773.316841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2778.594055] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2778.633083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2779.630764] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2779.678336] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2779.680487] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2779.699818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2781.220755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2781.365839] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2781.512529] Lustre: lustre-OST0001: new disk, initializing [ 2781.523544] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2781.531623] Lustre: 100800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2781.952171] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2781.962374] Lustre: Skipped 2 previous similar messages [ 2789.334713] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2789.987943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2790.007168] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2790.183262] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2797.862581] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2804.798322] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2810.361781] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2810.369925] LustreError: Skipped 1 previous similar message [ 2810.379244] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.424256] Lustre: Skipped 1 previous similar message [ 2810.445445] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2816.788912] Lustre: server umount lustre-MDT0001 complete [ 2818.260660] LustreError: 98526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.290614] LustreError: 98526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2828.702714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2828.858573] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2828.868682] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2828.958799] Lustre: 98521:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2831.037549] LustreError: 102327:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2831.057526] LustreError: 102327:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2831.076942] LustreError: 98523:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2831.290517] LustreError: 102339:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.135@tcp: cfg command failed: rc = -17 [ 2831.305866] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2831.305866] [ 2835.446834] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2836.026383] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2847.084919] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:53:31 (1780998811) [ 2865.958021] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2883.808850] Lustre: 104369:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2909.858716] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2914.303643] Lustre: 105503:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2998.240227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2998.249438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.263706] Lustre: Skipped 2 previous similar messages [ 2998.267916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2998.275147] Lustre: Skipped 6 previous similar messages [ 3003.916887] Lustre: server umount lustre-MDT0000 complete [ 3004.386760] LustreError: 98532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.409063] LustreError: 98532:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3007.860618] LustreError: 98511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780998974 with bad export cookie 10088382497738036524 [ 3007.863495] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.873316] LustreError: 98511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3008.303801] Lustre: server umount lustre-MDT0001 complete [ 3023.625214] Lustre: server umount lustre-OST0000 complete [ 3025.378555] Lustre: 88714:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780998975/real 1780998975] req@ffff9319834f4000 x1867512169205760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780998991 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3025.403637] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.420936] Lustre: Skipped 3 previous similar messages [ 3028.161853] Lustre: server umount lustre-OST0001 complete [ 3035.203348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3047.115965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3057.002708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3067.484176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3088.718962] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 3098.887850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3098.929411] Lustre: lustre-MDT0000: reset Object Index mappings [ 3099.689127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.695540] Lustre: Skipped 1 previous similar message [ 3103.919755] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3111.453766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.475076] Lustre: lustre-MDT0001: reset Object Index mappings [ 3116.658880] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3120.322822] Lustre: 110549:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3127.103479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3134.674131] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3136.295407] LustreError: 110902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.325273] LustreError: 110902:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3136.333694] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3143.415633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3145.164993] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3145.185805] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3145.294552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3150.464712] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3158.568406] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3162.417568] Lustre: 112394:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3219.041976] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:59:43 (1780999183) [ 3247.632466] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.646502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.656911] Lustre: Skipped 2 previous similar messages [ 3247.677376] Lustre: Skipped 6 previous similar messages [ 3251.700631] Lustre: server umount lustre-MDT0000 complete [ 3255.487111] LustreError: 109418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780999221 with bad export cookie 10088382497738083620 [ 3255.507167] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3262.042445] Lustre: server umount lustre-MDT0001 complete [ 3272.729798] Lustre: server umount lustre-OST0000 complete [ 3291.103223] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3291.332290] Lustre: server umount lustre-OST0001 complete [ 3299.211714] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 3309.128907] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 3351.516916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.879655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3351.937649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3352.088322] Lustre: lustre-MDT0000: new disk, initializing [ 3352.263252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3352.277923] Lustre: Skipped 3 previous similar messages [ 3352.333698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3358.516302] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3366.962622] Lustre: server umount lustre-MDT0000 complete [ 3377.140507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.457679] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3377.478516] Lustre: 116625:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3377.498524] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3382.865623] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3388.092773] Lustre: server umount lustre-MDT0000 complete [ 3396.902631] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 06:02:41 (1780999361) [ 3398.442485] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3400.204718] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 06:02:45 (1780999365) [ 3417.506760] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 3426.806353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3431.370359] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3440.174854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3440.350571] Lustre: 118641:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3440.429883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3440.436473] Lustre: Skipped 1 previous similar message [ 3440.499974] Lustre: lustre-MDT0001: new disk, initializing [ 3440.662325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3440.674672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3444.551367] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3448.722499] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3456.116353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3456.562405] Lustre: lustre-OST0000: new disk, initializing [ 3456.566232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3456.574743] Lustre: 120185:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3458.626068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3458.642467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3458.823719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3463.665630] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3474.850261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3474.952958] Lustre: lustre-OST0001: new disk, initializing [ 3474.956341] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3474.961588] Lustre: 121037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3476.190016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3476.213225] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3476.413040] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3480.777192] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3490.412623] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3494.500852] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3868.010352] Lustre: 127449:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3868.021463] Lustre: 127449:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3868.031783] Lustre: 127449:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3868.044544] Lustre: 127449:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3868.057532] Lustre: 127449:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3868.067107] Lustre: 127449:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4176.704289] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 06:15:40 (1781000140) [ 4193.253798] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4193.262280] Lustre: Skipped 6 previous similar messages [ 4193.267879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4193.277533] Lustre: Skipped 7 previous similar messages [ 4198.372228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4198.375119] Lustre: Skipped 3 previous similar messages [ 4199.644739] Lustre: server umount lustre-MDT0000 complete [ 4203.491863] LustreError: 121845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4203.521640] LustreError: 121845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 4203.914712] LustreError: 118633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781000170 with bad export cookie 10088382497738153333 [ 4203.929096] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4203.946885] LustreError: 118633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4208.616490] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.623905] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4208.631703] Lustre: Skipped 1 previous similar message [ 4208.654101] Lustre: Skipped 1 previous similar message [ 4218.335117] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4218.805604] Lustre: server umount lustre-MDT0001 complete [ 4230.758532] Lustre: server umount lustre-OST0000 complete [ 4240.965473] Lustre: server umount lustre-OST0001 complete [ 4248.468205] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 4261.150439] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 4298.055485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4298.286949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4298.310255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4298.410912] Lustre: lustre-MDT0000: new disk, initializing [ 4298.523159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4298.532642] Lustre: Skipped 5 previous similar messages [ 4298.557916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4302.849922] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4313.148351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4313.353286] Lustre: 130475:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4313.437266] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4313.440194] Lustre: Skipped 1 previous similar message [ 4313.547392] Lustre: lustre-MDT0001: new disk, initializing [ 4313.741925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4313.752990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4318.613668] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4326.394491] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4330.306566] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4338.481058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4338.906708] Lustre: lustre-OST0000: new disk, initializing [ 4338.910336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4338.917411] Lustre: 132218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4339.022673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4339.031116] Lustre: Skipped 1 previous similar message [ 4340.193367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4340.227447] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4340.397153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4347.803083] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4357.342381] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4369.560714] Lustre: lustre-MDD0000: changelog on [ 4373.107543] Lustre: lustre-MDD0001: changelog on [ 6261.166071] Lustre: lustre-MDD0001: changelog off [ 6263.663632] Lustre: 130483:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6264.900890] Lustre: lustre-MDD0000: changelog off [ 6275.554460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6275.559430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6275.580471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6275.588680] Lustre: Skipped 1 previous similar message [ 6275.596331] Lustre: Skipped 4 previous similar messages [ 6277.970540] Lustre: server umount lustre-OST0000 complete [ 6284.769078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6284.779478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6284.798045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6284.807312] Lustre: Skipped 1 previous similar message [ 6285.802416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.468978] Lustre: server umount lustre-MDT0000 complete [ 6290.917635] LustreError: 131316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6290.935949] LustreError: 131316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6293.914801] LustreError: 130467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781002260 with bad export cookie 10088382497738973054 [ 6293.916180] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6293.924250] LustreError: 130467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6294.427884] Lustre: server umount lustre-MDT0001 complete [ 6309.325433] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 6313.028873] Key type lgssc unregistered [ 6313.408677] LNet: 141226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6313.420782] LNetError: 141226:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6313.459819] LNet: Removed LNI 192.168.203.135@tcp [ 6314.656249] Key type .llcrypt unregistered [ 6314.661249] Key type ._llcrypt unregistered [ 6338.773256] Key type ._llcrypt registered [ 6338.775905] Key type .llcrypt registered [ 6338.896760] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 6357.002153] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 6358.390256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6358.441103] alg: No test for adler32 (adler32-zlib) [ 6359.568512] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 6359.816272] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 6361.503472] Key type lgssc registered [ 6362.752655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6404.465337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6404.498535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.828759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6405.880851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6406.032435] Lustre: lustre-MDT0000: new disk, initializing [ 6406.203284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6406.236304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6411.787508] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6423.511250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6423.682512] Lustre: 144396:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6423.751778] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6423.757372] Lustre: Skipped 1 previous similar message [ 6423.853581] Lustre: lustre-MDT0001: new disk, initializing [ 6423.934174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6423.970611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6423.980814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6428.215105] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6436.392475] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6440.685389] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6447.470211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6447.815906] Lustre: lustre-OST0000: new disk, initializing [ 6447.819763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6447.825237] Lustre: 146140:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6447.956950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6453.838434] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6455.336055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6455.345716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6455.420117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6460.996659] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6466.255545] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_module obdecho/obdecho [ 6468.195515] LustreError: 147013:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6468.207990] LustreError: 147013:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6470.027772] LustreError: 147109:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6485.483545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6485.498592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6485.526033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6489.411628] Lustre: server umount lustre-MDT0000 complete [ 6494.228533] LustreError: 144391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781002460 with bad export cookie 3910434586161929382 [ 6494.235208] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6494.252712] LustreError: 144391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6494.764397] Lustre: server umount lustre-MDT0001 complete [ 6509.738636] Lustre: server umount lustre-OST0000 complete [ 6531.333740] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 6543.118849] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 6582.430351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6582.858388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6582.896542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6583.004577] Lustre: lustre-MDT0000: new disk, initializing [ 6583.133287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.173434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6588.741889] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6599.525086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6599.660445] Lustre: 150795:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6599.740775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6599.757799] Lustre: Skipped 1 previous similar message [ 6599.886284] Lustre: lustre-MDT0001: new disk, initializing [ 6600.021979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6600.063368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6600.078063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6605.285766] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6612.945739] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6616.693589] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6624.052992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6624.388740] Lustre: lustre-OST0000: new disk, initializing [ 6624.393480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6624.401358] Lustre: 152536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6624.489769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6630.496074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6630.521383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6630.629014] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6630.991705] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6639.802139] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6644.928582] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6645.178350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6650.420540] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6650.673264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6656.481758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6656.487991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6656.494492] Lustre: Skipped 2 previous similar messages [ 6656.497491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6656.497499] Lustre: Skipped 2 previous similar messages [ 6658.930304] Lustre: server umount lustre-OST0000 complete [ 6666.721907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6666.724184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6666.734098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6666.734106] Lustre: Skipped 1 previous similar message [ 6666.737082] Lustre: Skipped 2 previous similar messages [ 6669.067382] Lustre: server umount lustre-MDT0000 complete [ 6671.841952] LustreError: 150802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6671.858783] LustreError: 150802:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6673.368188] LustreError: 150786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781002639 with bad export cookie 3910434586161930341 [ 6673.381343] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6673.391307] LustreError: 150786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6674.253907] Lustre: server umount lustre-MDT0001 complete [ 6678.190816] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:57:22 (1781002642) [ 6679.967955] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6681.739154] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:57:26 (1781002646) [ 6689.402058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6689.878963] LustreError: 154803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6689.982523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6694.558586] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6695.395311] LustreError: 154804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6700.521927] LustreError: 154803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6704.565590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6710.668902] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6717.745918] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6722.085963] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6729.632735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6730.208856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6730.220304] Lustre: Skipped 1 previous similar message [ 6736.983567] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6744.139766] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6755.571202] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping) [ 6755.589744] Lustre: Skipped 1 previous similar message [ 6755.811225] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6755.831464] Lustre: Skipped 1 previous similar message [ 6757.281339] Lustre: server umount lustre-MDT0000 complete [ 6760.665375] LustreError: 155912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6762.017557] LustreError: 154784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781002728 with bad export cookie 3910434586161931223 [ 6762.027776] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6762.052336] LustreError: 154784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6765.781879] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.35@tcp (stopping) [ 6765.784628] Lustre: Skipped 3 previous similar messages [ 6766.050898] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6766.074541] Lustre: Skipped 1 previous similar message [ 6768.724212] Lustre: server umount lustre-MDT0001 complete [ 6775.664743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6790.929221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6791.248929] LustreError: 158307:0:(llog.c:1646:llog_backup()) MGC192.168.203.135@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6791.255977] Lustre: 158307:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.135@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6792.673566] LustreError: 158312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6792.700754] LustreError: 158312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6811.345250] LustreError: 158314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6811.365980] LustreError: 158314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6843.874631] LustreError: 158314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6843.892136] LustreError: 158314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 6852.579237] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6852.584764] Lustre: 158307:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.135@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6893.557985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6898.659899] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6899.314538] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6907.878402] LustreError: 158342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6907.914877] LustreError: 158342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 6909.508708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.234311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6939.123074] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6939.638909] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6947.758388] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6949.897569] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6954.373759] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6956.305872] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6974.945928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.948890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6974.973412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6974.973419] Lustre: Skipped 2 previous similar messages [ 6981.042924] Lustre: server umount lustre-MDT0000 complete [ 6984.894720] LustreError: 157248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781002951 with bad export cookie 3910434586161937712 [ 6984.900458] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6984.905377] LustreError: 157248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6985.543982] Lustre: server umount lustre-MDT0001 complete [ 6993.138942] Lustre: server umount lustre-OST0000 complete [ 7016.644936] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 7028.035894] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7067.057300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7067.354152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7067.390215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7067.491819] Lustre: lustre-MDT0000: new disk, initializing [ 7067.619042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7067.623672] Lustre: Skipped 1 previous similar message [ 7067.654271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7072.287249] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7084.783202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7084.955995] Lustre: 163168:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7085.281188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7091.117837] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7099.999957] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7104.418666] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7111.414238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7111.857302] Lustre: lustre-OST0000: new disk, initializing [ 7111.863441] Lustre: Skipped 1 previous similar message [ 7111.868448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7111.877077] Lustre: Skipped 2 previous similar messages [ 7111.891903] Lustre: 164910:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7117.851644] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7121.040925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7121.054470] Lustre: Skipped 1 previous similar message [ 7121.066051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7121.199959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7126.038685] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7130.404959] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7130.610526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7135.642305] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7135.895254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7141.856203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7141.859776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.864487] LustreError: Skipped 1 previous similar message [ 7141.865266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7141.865270] Lustre: Skipped 6 previous similar messages [ 7141.913112] Lustre: Skipped 4 previous similar messages [ 7144.133735] Lustre: server umount lustre-OST0000 complete [ 7154.323678] Lustre: server umount lustre-MDT0000 complete [ 7157.220264] LustreError: 163175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7157.233130] LustreError: 163175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 7158.291345] LustreError: 163160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781003124 with bad export cookie 3910434586161954505 [ 7158.293977] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7158.309157] LustreError: 163160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7158.645770] Lustre: server umount lustre-MDT0001 complete [ 7167.450904] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 07:05:32 (1781003132) [ 7173.540163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7177.930757] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7184.976096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.943712] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7194.887977] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7198.638634] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7205.603719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7215.175926] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7223.057370] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7235.363236] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 07:06:39 (1781003199) [ 7240.164119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7240.166340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7240.169474] LustreError: Skipped 1 previous similar message [ 7240.171340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7240.171347] Lustre: Skipped 4 previous similar messages [ 7240.174485] Lustre: Skipped 3 previous similar messages [ 7245.995963] Lustre: server umount lustre-OST0000 complete [ 7256.318805] Lustre: server umount lustre-MDT0000 complete [ 7260.350208] LustreError: 169809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781003226 with bad export cookie 3910434586161955394 [ 7260.358247] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7260.366538] LustreError: 169809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7260.645610] Lustre: server umount lustre-MDT0001 complete [ 7272.032929] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 7276.173728] Key type lgssc unregistered [ 7276.415888] LNet: 170544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7276.419971] LNetError: 170544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7276.429915] LNet: Removed LNI 192.168.203.135@tcp [ 7277.247779] Key type .llcrypt unregistered [ 7277.254697] Key type ._llcrypt unregistered [ 7303.666352] Key type ._llcrypt registered [ 7303.672431] Key type .llcrypt registered [ 7303.800928] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7305.129980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7305.187344] alg: No test for adler32 (adler32-zlib) [ 7306.452089] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 7306.765359] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 7308.527187] Key type lgssc registered [ 7309.752637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7317.311508] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7317.321762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7332.831306] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7337.951200] LustreError: 171771:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.135@tcp: failed processing log, type 4: rc = -110 [ 7343.072413] LustreError: 171771:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7343.091979] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7343.106463] LustreError: MGC192.168.203.135@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7343.115908] LustreError: 171771:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7343.121504] LustreError: 171771:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 7343.125972] LustreError: 171771:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7343.211383] Lustre: server umount lustre-OST0000 complete [ 7343.217843] LustreError: 171771:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7351.273408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7352.743301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.026838] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7362.029545] LustreError: 172272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7362.053490] LustreError: 172272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7364.785064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7365.138729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7369.455715] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7375.055862] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7378.769954] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7385.362794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7385.586861] LustreError: 173863:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7385.597663] Lustre: 173863:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.135@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7385.743449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7392.011610] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7397.405165] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7400.930306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7400.938870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7400.967518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7402.498293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7402.501433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7402.522218] Lustre: Skipped 1 previous similar message [ 7402.523576] Lustre: Skipped 1 previous similar message [ 7406.032825] Lustre: server umount lustre-MDT0000 complete [ 7407.591487] LustreError: 172271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7407.622618] LustreError: 172271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7409.377673] LustreError: 172296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781003375 with bad export cookie 5972192919869306074 [ 7409.380248] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7409.388340] LustreError: 172296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7409.748577] Lustre: server umount lustre-MDT0001 complete [ 7423.586613] Lustre: server umount lustre-OST0000 complete [ 7432.615726] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 7440.873469] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7470.625145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7470.827964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7470.868237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7470.948226] Lustre: lustre-MDT0000: new disk, initializing [ 7471.011418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7471.033383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7474.729324] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7483.496777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7483.573340] Lustre: 177239:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7483.629463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7483.634908] Lustre: Skipped 1 previous similar message [ 7483.709291] Lustre: lustre-MDT0001: new disk, initializing [ 7483.812714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7483.856799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7483.864990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7487.282674] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7493.213445] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7496.816443] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7503.300939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7503.530987] Lustre: lustre-OST0000: new disk, initializing [ 7503.534943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7503.543962] Lustre: 178979:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7503.603955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7509.944774] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7510.129754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7510.146125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7510.330825] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7518.045830] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7523.025571] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7523.230514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7527.562794] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7527.738461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7530.976087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7530.979253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7530.993420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7531.010475] Lustre: Skipped 1 previous similar message [ 7531.023425] Lustre: Skipped 2 previous similar messages [ 7535.810873] Lustre: server umount lustre-OST0000 complete [ 7540.199577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7540.210334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7540.232295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7545.976910] Lustre: server umount lustre-MDT0000 complete [ 7546.344432] LustreError: 177246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7546.358069] LustreError: 177246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7549.995128] LustreError: 178024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781003516 with bad export cookie 5972192919869306795 [ 7549.997734] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7550.007351] LustreError: 178024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7550.348299] Lustre: server umount lustre-MDT0001 complete [ 7558.476862] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 07:12:03 (1781003523) [ 7574.982639] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 7577.373363] Key type lgssc unregistered [ 7577.650187] LNet: 181759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7577.660415] LNetError: 181759:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7577.676550] LNet: Removed LNI 192.168.203.135@tcp [ 7578.316944] Key type .llcrypt unregistered [ 7578.319515] Key type ._llcrypt unregistered [ 7593.983446] Key type ._llcrypt registered [ 7593.991093] Key type .llcrypt registered [ 7594.084847] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7595.287428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7595.426417] alg: No test for adler32 (adler32-zlib) [ 7596.488394] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 7596.698500] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 7598.343260] Key type lgssc registered [ 7599.384124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7606.818279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7606.837360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7608.313307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7612.257385] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7618.534488] LustreError: 182962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7618.553750] LustreError: 182962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7619.676852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7619.954719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7624.068526] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7629.638871] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7633.122558] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7640.214690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7640.382671] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7640.392489] LustreError: 184599:0:(llog.c:1687:llog_backup()) MGC192.168.203.135@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7640.411383] Lustre: 184599:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.135@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7640.541522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7647.531424] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7652.509582] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7655.905136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7655.918702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.935664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7657.960155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7657.968111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7657.999816] Lustre: Skipped 2 previous similar messages [ 7661.630841] Lustre: server umount lustre-MDT0000 complete [ 7663.083800] LustreError: 184121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7663.101084] LustreError: 184121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7664.750242] LustreError: 182942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781003631 with bad export cookie 16922463683324948163 [ 7664.755925] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7664.760657] LustreError: 182942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7665.162480] Lustre: server umount lustre-MDT0001 complete [ 7678.776339] Lustre: server umount lustre-OST0000 complete [ 7686.459248] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 7695.040228] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7732.265761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7732.578249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7732.663586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7732.791986] Lustre: lustre-MDT0000: new disk, initializing [ 7732.922385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.955703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7737.152391] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7747.368960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7747.548766] Lustre: 187978:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7747.594271] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7747.596961] Lustre: Skipped 1 previous similar message [ 7747.600700] LustreError: 188744:0:(llog.c:1687:llog_backup()) MGC192.168.203.135@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7747.607518] LustreError: 188744:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7747.612314] Lustre: 188744:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.135@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7747.630305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7747.635227] Lustre: Skipped 1 previous similar message [ 7747.789778] Lustre: lustre-MDT0001: new disk, initializing [ 7747.948284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7748.003180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7748.023870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7751.984673] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7758.917400] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7763.713342] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7771.839703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7772.209436] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7772.214854] Lustre: Skipped 1 previous similar message [ 7772.219591] LustreError: 189696:0:(llog.c:1687:llog_backup()) MGC192.168.203.135@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7772.231706] LustreError: 189696:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7772.244314] Lustre: 189696:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.135@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7772.260257] Lustre: 189696:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7772.285664] Lustre: lustre-OST0000: new disk, initializing [ 7772.291320] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7772.300693] Lustre: 189721:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7772.434633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7778.981266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7778.999696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7779.149429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7780.685528] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7790.765322] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7795.569063] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7795.766581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7799.943807] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7800.178468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7804.897437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7804.898543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7804.907846] LustreError: Skipped 1 previous similar message [ 7804.911247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7804.941463] Lustre: Skipped 2 previous similar messages [ 7810.018479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7810.037414] Lustre: Skipped 2 previous similar messages [ 7815.138980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7815.152570] Lustre: Skipped 1 previous similar message [ 7816.159123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7816.357512] Lustre: server umount lustre-OST0000 complete [ 7820.263279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7820.279138] Lustre: Skipped 1 previous similar message [ 7825.376437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7825.390373] Lustre: Skipped 3 previous similar messages [ 7826.319941] Lustre: server umount lustre-MDT0000 complete [ 7830.336207] LustreError: 187970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781003796 with bad export cookie 16922463683324948877 [ 7830.342148] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7830.344458] LustreError: 187970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7830.879072] Lustre: server umount lustre-MDT0001 complete [ 7841.562392] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 07:16:45 (1781003805) [ 7857.530904] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7867.003152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7867.326956] LustreError: 192686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7867.361354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7870.930808] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7872.482566] LustreError: 192687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7877.601767] LustreError: 192686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.434580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.553552] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7885.094467] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7892.008925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7898.223226] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7905.788645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7905.931339] Lustre: lustre-OST0001: new disk, initializing [ 7905.934737] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7905.944426] Lustre: 194941:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7906.028773] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7906.032767] Lustre: Skipped 2 previous similar messages [ 7911.708841] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7914.618605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7914.635211] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7914.782948] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7923.588946] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7930.452116] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7932.281752] Lustre: DEBUG MARKER: ADD OST3 [ 7936.980409] loop: module loaded [ 7937.039491] loop0: detected capacity change from 0 to 204800000 [ 7937.289438] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7937.299762] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7937.311503] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7943.404022] loop0: detected capacity change from 0 to 204800000 [ 7944.369377] Lustre: DEBUG MARKER: STOP OST3 [ 7949.670497] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7949.857200] Lustre: lustre-OST0003: new disk, initializing [ 7949.860889] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7949.873099] Lustre: 196915:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7951.316773] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7951.430304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7951.441111] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7951.473140] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7953.375651] LustreError: 197080:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7953.776089] Lustre: Failing over lustre-OST0003 [ 7953.910241] Lustre: server umount lustre-OST0003 complete [ 7956.101212] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7956.466047] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7956.471174] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7956.480281] LustreError: 194136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7956.496291] Lustre: Skipped 1 previous similar message [ 7956.520528] LustreError: 194136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7965.244688] Lustre: DEBUG MARKER: START OST3 again [ 7971.144671] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7971.439881] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7971.449623] Lustre: Skipped 1 previous similar message [ 7971.468150] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7972.037129] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7972.828748] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7972.834264] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7977.003852] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7997.749384] Lustre: Failing over lustre-OST0003 [ 7997.927945] Lustre: server umount lustre-OST0003 complete [ 7998.432680] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7998.437401] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7998.441831] LustreError: Skipped 1 previous similar message [ 7998.463520] Lustre: Skipped 1 previous similar message [ 7998.476618] LustreError: 195396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7998.491362] LustreError: 195396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 8017.382604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8017.389556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.394840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8017.410014] Lustre: Skipped 2 previous similar messages [ 8017.424190] Lustre: Skipped 3 previous similar messages [ 8022.401819] Lustre: server umount lustre-MDT0000 complete [ 8026.144355] LustreError: 192668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781003992 with bad export cookie 16922463683324949766 [ 8026.184911] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8031.653635] LustreError: 192682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8031.686491] LustreError: 192682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 8032.936396] Lustre: server umount lustre-MDT0001 complete [ 8036.825507] Lustre: server umount lustre-OST0000 complete [ 8040.786962] Lustre: server umount lustre-OST0001 complete [ 8047.037503] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 8054.218406] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8094.083610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8094.410945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8094.468823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8094.594669] Lustre: lustre-MDT0000: new disk, initializing [ 8094.703481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8099.531538] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8109.552403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8109.672689] Lustre: 202319:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8109.981886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8109.986821] Lustre: Skipped 1 previous similar message [ 8110.016425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8113.628514] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8120.128980] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8123.525523] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8129.756211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8130.075110] Lustre: lustre-OST0000: new disk, initializing [ 8130.080024] Lustre: Skipped 1 previous similar message [ 8130.085553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8130.094664] Lustre: Skipped 2 previous similar messages [ 8130.103884] Lustre: 204062:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8131.526722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8131.534306] Lustre: Skipped 1 previous similar message [ 8131.538275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8131.672359] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8135.970560] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8142.505335] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8146.663493] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8146.822643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8151.531907] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8151.770367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8157.153580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8157.154519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8157.166433] LustreError: Skipped 1 previous similar message [ 8157.192973] Lustre: Skipped 4 previous similar messages [ 8157.208936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8157.215695] Lustre: Skipped 5 previous similar messages [ 8159.916715] Lustre: server umount lustre-OST0000 complete [ 8170.050995] Lustre: server umount lustre-MDT0000 complete [ 8172.514346] LustreError: 202331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8172.544455] LustreError: 202331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8174.616042] LustreError: 202312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781004140 with bad export cookie 16922463683324952062 [ 8174.622611] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8174.634546] LustreError: 202312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8175.097549] Lustre: server umount lustre-MDT0001 complete [ 8179.261524] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 07:22:24 (1781004144) [ 8196.632249] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 8208.706416] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8248.922694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8249.167656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8249.203760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8249.266690] Lustre: lustre-MDT0000: new disk, initializing [ 8249.339673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8253.682960] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8264.303339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8264.438727] Lustre: 208489:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8264.877389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8269.039209] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8276.301213] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8280.426550] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8286.593358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8286.854664] Lustre: 210231:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8288.747470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8293.108178] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8300.856644] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8305.306384] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8305.539468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8309.871848] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8310.074398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8314.338750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8314.350075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8314.354956] Lustre: Skipped 2 previous similar messages [ 8314.367751] LustreError: Skipped 1 previous similar message [ 8314.369536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8314.369542] Lustre: Skipped 3 previous similar messages [ 8317.870923] Lustre: server umount lustre-OST0000 complete [ 8327.449339] Lustre: server umount lustre-MDT0000 complete [ 8329.698813] LustreError: 208498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8329.712895] LustreError: 208498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8331.146083] LustreError: 208483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781004297 with bad export cookie 16922463683324952944 [ 8331.157779] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8331.550157] Lustre: server umount lustre-MDT0001 complete [ 8339.798419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8344.075109] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8353.004094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8357.047958] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8361.988684] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8365.252286] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8370.994562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8371.248358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8371.252935] Lustre: Skipped 6 previous similar messages [ 8376.278697] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8380.887724] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8398.819367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8398.835658] LustreError: Skipped 1 previous similar message [ 8402.769830] Lustre: server umount lustre-OST0000 complete [ 8412.817444] Lustre: server umount lustre-MDT0000 complete [ 8416.268152] LustreError: 212285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781004382 with bad export cookie 16922463683324953826 [ 8416.269164] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8416.276483] LustreError: 212285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8416.564095] Lustre: server umount lustre-MDT0001 complete [ 8521.231867] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 8523.900534] Key type lgssc unregistered [ 8524.218873] LNet: 215627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8524.236228] LNetError: 215627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8524.251605] LNet: Removed LNI 192.168.203.135@tcp [ 8524.992807] Key type .llcrypt unregistered [ 8524.998548] Key type ._llcrypt unregistered [ 8544.900533] Key type ._llcrypt registered [ 8544.903536] Key type .llcrypt registered [ 8545.009381] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 8557.968326] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8558.895301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8558.921515] alg: No test for adler32 (adler32-zlib) [ 8560.005769] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 8560.246087] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 8561.975164] Key type lgssc registered [ 8562.985334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8593.419930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8593.459542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8594.694890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8594.718279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8594.787257] Lustre: lustre-MDT0000: new disk, initializing [ 8594.910612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8594.924433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8598.299993] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8607.289626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8607.432420] Lustre: 218923:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8607.517409] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8607.522518] Lustre: Skipped 1 previous similar message [ 8607.609431] Lustre: lustre-MDT0001: new disk, initializing [ 8607.722421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8607.752490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8607.763157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8611.410650] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8617.854198] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8621.239731] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8626.807568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8627.074195] Lustre: lustre-OST0000: new disk, initializing [ 8627.079246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8627.087199] Lustre: 220665:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8627.178409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.403749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8632.424769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8632.537165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8632.791495] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8640.034199] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8644.561775] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8644.781982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8649.214547] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8649.455840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8653.282022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8653.284611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8653.286046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8653.297916] Lustre: Skipped 1 previous similar message [ 8657.310685] Lustre: server umount lustre-OST0000 complete [ 8663.526586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8663.545480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8663.556559] Lustre: Skipped 2 previous similar messages [ 8668.643277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8673.762385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8673.786246] Lustre: Skipped 2 previous similar messages [ 8675.295126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8675.530913] Lustre: server umount lustre-MDT0000 complete [ 8678.909350] LustreError: 218915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781004645 with bad export cookie 21602912822797210 [ 8678.910949] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8678.924231] LustreError: 218915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8679.134558] Lustre: server umount lustre-MDT0001 complete [ 8695.084888] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8703.847604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8704.314811] LustreError: 223437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8704.330342] LustreError: 223437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8704.392880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8708.227521] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8709.604406] LustreError: 223438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.722147] LustreError: 223437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.975565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8715.297978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8718.278668] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8720.179790] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8725.371953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8725.481169] LustreError: 224884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8725.629851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8731.396246] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8739.115132] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8739.271903] Lustre: lustre-OST0001: new disk, initializing [ 8739.275122] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8739.282550] Lustre: 225692:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8744.903744] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8745.620366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8745.630696] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8745.786740] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8755.305499] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8764.257385] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8776.213376] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 07:32:21 (1781004741) [ 8781.794500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8781.800686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8781.803115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8781.810790] Lustre: Skipped 2 previous similar messages [ 8783.967193] Lustre: server umount lustre-MDT0000 complete [ 8786.923350] LustreError: 223438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8786.944258] LustreError: 223438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8787.077983] LustreError: 226443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781004753 with bad export cookie 21602912822798092 [ 8787.081378] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8787.093299] LustreError: 226443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8787.642822] Lustre: server umount lustre-MDT0001 complete [ 8801.397235] Lustre: server umount lustre-OST0000 complete [ 8816.097508] Lustre: server umount lustre-OST0001 complete [ 8822.795818] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 8832.133188] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8863.997133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8864.216776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8864.260660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8864.366374] Lustre: lustre-MDT0000: new disk, initializing [ 8864.462481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8864.469475] Lustre: Skipped 1 previous similar message [ 8864.485416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8868.587450] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8878.499042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8878.641354] Lustre: 229586:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8878.684510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8878.687953] Lustre: Skipped 1 previous similar message [ 8878.749901] Lustre: lustre-MDT0001: new disk, initializing [ 8878.844621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8878.852570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8883.000318] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8890.031907] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8893.817781] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8899.521598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8899.813925] Lustre: lustre-OST0000: new disk, initializing [ 8899.818893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8899.828198] Lustre: 231329:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8899.914299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8899.920368] Lustre: Skipped 1 previous similar message [ 8901.341473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8901.367479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8901.660961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8906.707916] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8915.193290] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8919.574072] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8919.779646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8923.800363] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8923.985816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8927.201987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8927.204622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8927.225001] LustreError: Skipped 1 previous similar message [ 8927.225732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8927.225737] Lustre: Skipped 3 previous similar messages [ 8927.273801] Lustre: Skipped 3 previous similar messages [ 8932.050405] Lustre: server umount lustre-OST0000 complete [ 8937.443279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8937.485108] Lustre: Skipped 1 previous similar message [ 8943.157750] Lustre: server umount lustre-MDT0000 complete [ 8947.015241] LustreError: 231328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781004913 with bad export cookie 21602912822800220 [ 8947.019404] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8947.043516] LustreError: 231328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8947.414520] Lustre: server umount lustre-MDT0001 complete [ 8955.068297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8955.538418] LustreError: 233400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8959.545502] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8967.501629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.813784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8967.820601] Lustre: Skipped 1 previous similar message [ 8972.432624] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8977.389858] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8980.688452] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8986.934684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8993.611945] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8999.426561] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9146.337148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9146.347668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9146.372852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9146.381554] Lustre: Skipped 5 previous similar messages [ 9160.159463] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9160.285498] Lustre: server umount lustre-OST0000 complete [ 9164.778387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9164.791071] Lustre: Skipped 2 previous similar messages [ 9169.266287] Lustre: server umount lustre-MDT0000 complete [ 9169.888795] LustreError: 233397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9169.904237] LustreError: 233397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9172.798227] LustreError: 233381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781005139 with bad export cookie 21602912822801109 [ 9172.805419] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9172.813903] LustreError: 233381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9173.192442] Lustre: server umount lustre-MDT0001 complete [ 9264.490401] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 9266.737941] Key type lgssc unregistered [ 9267.028561] LNet: 236648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9267.050119] LNetError: 236648:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9268.070179] LNet: Removed LNI 192.168.203.135@tcp [ 9268.727157] Key type .llcrypt unregistered [ 9268.729914] Key type ._llcrypt unregistered [ 9287.874607] Key type ._llcrypt registered [ 9287.877684] Key type .llcrypt registered [ 9287.985931] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 9299.283794] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9299.928957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9299.992035] alg: No test for adler32 (adler32-zlib) [ 9301.116271] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 9301.351043] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 9302.991198] Key type lgssc registered [ 9304.048573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9341.644593] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9351.547724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9351.569508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9352.974690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9353.042967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9353.164833] Lustre: lustre-MDT0000: new disk, initializing [ 9353.282409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9353.304326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9357.240912] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9366.379420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9366.530455] Lustre: 240646:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9366.591531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9366.595796] Lustre: Skipped 1 previous similar message [ 9366.706559] Lustre: lustre-MDT0001: new disk, initializing [ 9366.813341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9366.867765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9366.893799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9370.385773] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9374.639417] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9379.360219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9379.557160] Lustre: lustre-OST0000: new disk, initializing [ 9379.560760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9379.564164] Lustre: 242244:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9379.628459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9384.369715] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9387.099254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9387.107839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9387.238622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9393.442196] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9393.604358] Lustre: lustre-OST0001: new disk, initializing [ 9393.607932] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9393.615180] Lustre: 243098:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9393.697161] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9393.931455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9393.952105] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9394.045168] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9399.040696] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9409.791300] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9414.787694] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9424.867702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9424.873312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9424.892403] Lustre: Skipped 1 previous similar message [ 9429.985792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9430.003893] Lustre: Skipped 6 previous similar messages [ 9430.763130] Lustre: server umount lustre-MDT0000 complete [ 9433.906929] LustreError: 241872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781005400 with bad export cookie 339284989706315805 [ 9433.909414] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9433.917686] LustreError: 241872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9434.182245] Lustre: server umount lustre-MDT0001 complete [ 9447.491157] Lustre: server umount lustre-OST0000 complete [ 9460.851739] Lustre: server umount lustre-OST0001 complete [ 9474.143962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9483.270339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9492.590806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9501.557925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9519.225785] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9528.788766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9528.808504] Lustre: lustre-MDT0000: reset Object Index mappings [ 9529.347938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9532.595747] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9534.435519] LustreError: 248071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9534.449800] LustreError: 248071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9539.155649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9539.174493] Lustre: lustre-MDT0001: reset Object Index mappings [ 9539.536979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9542.749186] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9545.810780] Lustre: 249177:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9552.198092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9555.565493] LustreError: 249532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9555.582559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9557.646174] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9561.061692] LustreError: 249532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9564.129604] LustreError: 249532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9564.146795] LustreError: 249532:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9564.593527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9564.857846] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9564.860933] Lustre: Skipped 1 previous similar message [ 9570.306861] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9570.670573] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9577.057266] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9579.613554] Lustre: 251012:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9585.638734] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9585.639969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9585.645196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9585.650041] Lustre: Skipped 5 previous similar messages [ 9589.153501] Lustre: server umount lustre-MDT0000 complete [ 9590.756185] LustreError: 248066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9590.773444] LustreError: 248066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9591.879718] LustreError: 248050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781005558 with bad export cookie 339284989706322595 [ 9591.883811] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9591.894645] LustreError: 248050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9592.139249] Lustre: server umount lustre-MDT0001 complete [ 9605.174734] Lustre: server umount lustre-OST0000 complete [ 9619.280311] Lustre: server umount lustre-OST0001 complete [ 9637.616072] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:46:42 (1781005602) [ 9654.937854] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 9665.912578] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9698.418445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9698.688812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9698.738803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9698.840516] Lustre: lustre-MDT0000: new disk, initializing [ 9698.920549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9698.946748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9702.872325] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9712.124274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9712.221946] Lustre: 255235:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9712.296362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9712.301375] Lustre: Skipped 1 previous similar message [ 9712.381239] Lustre: lustre-MDT0001: new disk, initializing [ 9712.497333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9712.513744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9716.393038] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9723.140748] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9726.532493] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9732.938948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9733.187278] Lustre: lustre-OST0000: new disk, initializing [ 9733.191394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9733.205989] Lustre: 256976:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9734.576775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9734.596095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9734.716631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9740.231416] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9748.859567] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9754.560071] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9754.865936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9759.779257] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9759.972888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9765.347695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9765.362067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9765.387898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9765.399314] Lustre: Skipped 4 previous similar messages [ 9770.474304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9770.484657] Lustre: Skipped 1 previous similar message [ 9776.097325] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9776.313410] Lustre: server umount lustre-OST0000 complete [ 9780.706108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9780.716549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9780.720671] Lustre: Skipped 2 previous similar messages [ 9780.726516] Lustre: Skipped 2 previous similar messages [ 9786.748695] Lustre: server umount lustre-MDT0000 complete [ 9790.947198] LustreError: 255246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9790.965804] LustreError: 255246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9791.325355] LustreError: 255228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781005757 with bad export cookie 339284989706324688 [ 9791.327660] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9791.342927] LustreError: 255228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9791.981068] Lustre: server umount lustre-MDT0001 complete [ 9808.880575] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9818.375900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9818.894782] LustreError: 259747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9818.956787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9818.961536] Lustre: Skipped 2 previous similar messages [ 9823.025385] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9830.674306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9834.900778] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9837.372554] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9843.347326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9849.553564] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9857.801497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9857.992432] Lustre: lustre-OST0001: new disk, initializing [ 9857.996675] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9858.002136] Lustre: 262006:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9859.797549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9859.810588] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9859.904290] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9863.281407] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9872.507295] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9876.555450] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9881.892162] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9881.976489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9890.788253] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9890.792319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9890.799349] Lustre: Skipped 3 previous similar messages [ 9890.807435] Lustre: Skipped 6 previous similar messages [ 9895.007508] Lustre: server umount lustre-MDT0000 complete [ 9895.911494] LustreError: 263160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9895.930070] LustreError: 263160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9898.351918] LustreError: 262752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781005864 with bad export cookie 339284989706325570 [ 9898.357226] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9898.366605] LustreError: 262752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9898.665679] Lustre: server umount lustre-MDT0001 complete [ 9912.260829] Lustre: server umount lustre-OST0000 complete [ 9926.437886] Lustre: server umount lustre-OST0001 complete [ 9940.472943] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9949.845318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9950.277529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9950.284122] Lustre: Skipped 3 previous similar messages [ 9953.641179] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9960.419976] LustreError: 265030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9960.444602] LustreError: 265030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9960.843074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9964.545702] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9966.557114] Lustre: 266135:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9971.705987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9975.119052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9977.124457] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9983.813832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9986.046409] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9989.111403] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9995.327456] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9998.405996] Lustre: 267975:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10002.692135] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10002.754765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10010.302479] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:52:55 (1781005975) [10026.982691] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10026.985974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10027.003400] Lustre: Skipped 2 previous similar messages [10027.024041] Lustre: Skipped 3 previous similar messages [10030.497221] Lustre: server umount lustre-MDT0000 complete [10033.572625] LustreError: 265009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781005999 with bad export cookie 339284989706332388 [10033.575411] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10033.581786] LustreError: 265009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10040.027497] Lustre: server umount lustre-MDT0001 complete [10049.246253] Lustre: server umount lustre-OST0000 complete [10059.761840] Lustre: server umount lustre-OST0001 complete [10068.325657] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [10080.833914] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10124.925447] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10135.177713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10135.402724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10135.423862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10135.480852] Lustre: lustre-MDT0000: new disk, initializing [10135.594559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10140.091013] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10150.107571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10150.219987] Lustre: 272162:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10150.444186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10154.723956] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10165.178759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10165.429241] Lustre: 273763:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10167.286721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10171.160530] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10181.320621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10181.442963] Lustre: 274615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10183.354765] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10183.369149] Lustre: Skipped 1 previous similar message [10183.454481] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10187.049718] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10196.143783] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10199.696555] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10199.710135] Lustre: Skipped 1 previous similar message [10213.311091] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:56:18 (1781006178) [10216.937612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10216.953935] LustreError: Skipped 1 previous similar message [10216.966871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10216.988626] Lustre: Skipped 3 previous similar messages [10216.995410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10216.998092] Lustre: Skipped 4 previous similar messages [10217.709837] Lustre: server umount lustre-MDT0000 complete [10219.491743] LustreError: 272170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10219.515913] LustreError: 272170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [10220.862907] LustreError: 272941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781006187 with bad export cookie 339284989706339857 [10220.867569] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10220.875611] LustreError: 272941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10221.276414] Lustre: server umount lustre-MDT0001 complete [10235.361887] Lustre: server umount lustre-OST0000 complete [10249.604206] Lustre: server umount lustre-OST0001 complete [10257.009806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10257.460402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10257.468380] Lustre: Skipped 7 previous similar messages [10260.966313] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10267.949618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10271.164773] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10275.397857] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10277.775391] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10282.358346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10286.614868] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10290.607609] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10324.126761] loop1: detected capacity change from 0 to 204800000 [10324.379402] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10324.391406] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10324.404724] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10329.843592] loop1: detected capacity change from 0 to 204800000 [10336.098668] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10336.220840] LustreError: 276833:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs22490: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10336.238904] LustreError: 276833:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs22490 [10336.244378] LustreError: 276833:0:(mgs_handler.c:517:mgs_target_reg()) fs22490-MDT0001: Error checking target index: rc = -13 [10336.259836] LustreError: fs22490-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10336.283278] LustreError: 280241:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10336.303511] LustreError: 280241:0:(tgt_mount.c:2084:server_put_super()) no obd fs22490-MDT0001 [10336.316155] LustreError: 280241:0:(tgt_mount.c:116:server_deregister_mount()) fs22490-MDT0001 not registered [10336.362789] Lustre: server umount fs22490-MDT0001 complete [10336.367823] LustreError: 280241:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10344.177365] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10344.252547] Lustre: Setting parameter fs22490-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs22490-MDT0001 [10344.293800] Lustre: srv-fs22490-MDT0001: No data found on store. Initialize space. [10344.297744] Lustre: Skipped 4 previous similar messages [10344.369451] Lustre: fs22490-MDT0001: new disk, initializing [10344.378423] Lustre: Skipped 3 previous similar messages [10344.474985] Lustre: srv-fs22490-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10344.487548] Lustre: 280709:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22490-MDT0001-mdtlov: can't create new llog: rc = -115 [10344.497622] LustreError: 280709:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22490-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10348.599035] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10355.065502] Lustre: server umount fs22490-MDT0001 complete [10366.291308] loop1: detected capacity change from 0 to 204800000 [10371.085340] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10371.536191] Lustre: 282257:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22490-MDT0001-mdtlov: can't create new llog: rc = -115 [10371.549603] LustreError: 282257:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22490-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10375.141580] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10379.252672] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10380.147520] Lustre: server umount fs22490-MDT0001 complete [10384.357935] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10391.011679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10391.022402] Lustre: Skipped 3 previous similar messages [10391.027962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10405.343180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10405.694722] Lustre: server umount lustre-MDT0000 complete [10414.249465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10414.533710] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10420.019322] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10420.210603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10428.721769] loop1: detected capacity change from 0 to 204800000 [10435.218284] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10435.549410] Lustre: 284669:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22490-MDT0001-mdtlov: can't create new llog: rc = -115 [10435.566862] LustreError: 284669:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22490-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10440.022868] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10444.326608] Lustre: server umount fs22490-MDT0001 complete [10466.281351] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10466.285069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10466.289763] Lustre: Skipped 2 previous similar messages [10466.311184] LustreError: Skipped 1 previous similar message [10471.236797] Lustre: server umount lustre-OST0000 complete [10480.630694] Lustre: server umount lustre-MDT0000 complete [10481.645360] LustreError: 284205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10481.660928] LustreError: 284205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 104 previous similar messages [10484.657798] LustreError: 276824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781006450 with bad export cookie 339284989706342538 [10484.660205] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10484.670137] LustreError: 276824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10485.107321] Lustre: server umount lustre-MDT0001 complete [10496.200870] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [10499.763507] Key type lgssc unregistered [10500.244900] LNet: 286988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10500.254244] LNetError: 286988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10500.276431] LNet: Removed LNI 192.168.203.135@tcp [10501.145409] Key type .llcrypt unregistered [10501.147333] Key type ._llcrypt unregistered [10524.166608] Key type ._llcrypt registered [10524.172454] Key type .llcrypt registered [10524.306482] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10525.495079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10525.572461] alg: No test for adler32 (adler32-zlib) [10526.766176] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [10527.044874] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [10528.735222] Key type lgssc registered [10529.925581] Lustre: Echo OBD driver; http://www.lustre.org/ [10536.667705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10536.689738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10538.086893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10541.949784] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10548.206996] LustreError: 288246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10548.236480] LustreError: 288246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10550.307699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10550.741670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10554.958481] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10560.444972] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10563.988861] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10570.423551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10570.852817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10574.962727] LustreError: 289846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10577.101469] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10580.465300] LustreError: 289848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10582.516940] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10583.538463] LustreError: 289848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10583.574816] LustreError: 289848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10588.642805] LustreError: 289847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10588.662513] LustreError: 289847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10598.886607] LustreError: 289847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10598.907078] LustreError: 289847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10601.167891] loop1: detected capacity change from 0 to 204800000 [10601.332314] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10601.349224] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10601.369509] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10606.721104] loop1: detected capacity change from 0 to 204800000 [10612.599609] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10612.793632] Lustre: fs14602-OST0000: new disk, initializing [10612.797539] Lustre: srv-fs14602-OST0000: No data found on store. Initialize space. [10612.803041] Lustre: 291640:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10612.864135] Lustre: fs14602-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10616.011104] LustreError: 289848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10616.035075] LustreError: 289848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [10620.291701] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10631.152349] Lustre: server umount fs14602-OST0000 complete [10643.598874] loop1: detected capacity change from 0 to 204800000 [10648.368260] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10648.555420] Lustre: fs14602-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10650.086095] LustreError: 290072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10650.098867] LustreError: 290072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [10653.390790] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10660.331053] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10661.938425] loop2: detected capacity change from 0 to 204800000 [10662.199719] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10662.230089] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10662.239938] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10665.478419] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10668.622355] loop2: detected capacity change from 0 to 204800000 [10675.156190] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10675.272612] LustreError: fs14602-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10675.291983] LustreError: fs14602-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10675.314512] LustreError: 294645:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10675.333161] LustreError: 294645:0:(tgt_mount.c:2084:server_put_super()) no obd fs14602-OST0001 [10675.342648] LustreError: 294645:0:(tgt_mount.c:116:server_deregister_mount()) fs14602-OST0001 not registered [10675.377494] Lustre: server umount fs14602-OST0001 complete [10675.383793] LustreError: 294645:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10686.132405] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10686.315561] Lustre: fs14602-OST0001: new disk, initializing [10686.319928] Lustre: srv-fs14602-OST0001: No data found on store. Initialize space. [10686.327103] Lustre: 295197:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10686.445552] Lustre: fs14602-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10693.006731] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10702.021099] Lustre: server umount fs14602-OST0000 complete [10710.018864] Lustre: server umount fs14602-OST0001 complete [10716.645391] LustreError: 289846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10716.658095] LustreError: 289846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [10720.583849] loop1: detected capacity change from 0 to 204800000 [10724.823907] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10725.007654] Lustre: fs14602-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10729.710198] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10735.344065] loop2: detected capacity change from 0 to 204800000 [10740.156874] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10745.763724] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10753.458491] Lustre: server umount fs14602-OST0000 complete [10760.050393] Lustre: server umount fs14602-OST0001 complete [10769.889206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10769.896233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10769.911629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10772.962181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10772.966494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10772.968137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10772.978944] Lustre: Skipped 1 previous similar message [10774.618360] Lustre: server umount lustre-OST0000 complete [10781.154093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10781.162858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10781.175703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10783.207353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10783.212729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10783.235635] Lustre: Skipped 1 previous similar message [10784.559251] Lustre: server umount lustre-MDT0000 complete [10788.041149] LustreError: 295196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781006754 with bad export cookie 6907909966082840421 [10788.043068] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10788.054846] LustreError: 295196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10788.300120] Lustre: server umount lustre-MDT0001 complete [10797.023972] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [10799.639584] Key type lgssc unregistered [10799.919407] LNet: 301049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10799.929363] LNetError: 301049:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10799.950666] LNet: Removed LNI 192.168.203.135@tcp [10800.658129] Key type .llcrypt unregistered [10800.663815] Key type ._llcrypt unregistered [10826.205144] Key type ._llcrypt registered [10826.207101] Key type .llcrypt registered [10826.275138] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [10837.174569] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10837.859969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10837.891477] alg: No test for adler32 (adler32-zlib) [10838.897735] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [10839.079907] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [10840.736610] Key type lgssc registered [10841.566447] Lustre: Echo OBD driver; http://www.lustre.org/ [10867.856188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10867.870215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10869.192671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10869.237916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10869.366705] Lustre: lustre-MDT0000: new disk, initializing [10869.463644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10869.505368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10874.454188] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10885.270324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10885.378660] Lustre: 304853:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10885.466294] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10885.477053] Lustre: Skipped 1 previous similar message [10885.579225] Lustre: lustre-MDT0001: new disk, initializing [10885.660268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10885.727831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10885.736465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10889.455315] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10896.642272] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10900.339770] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10907.884520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10908.198264] Lustre: lustre-OST0000: new disk, initializing [10908.204944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10908.232278] Lustre: 306594:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10908.345472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10915.909199] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10915.955773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10915.975584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10916.123897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10924.177371] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10926.050235] Lustre: Mounted lustre-client [10926.816475] Lustre: Unmounted lustre-client [10936.301268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10936.312769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10936.345245] Lustre: Skipped 1 previous similar message [10941.415291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10941.419893] Lustre: Skipped 2 previous similar messages [10941.688448] Lustre: server umount lustre-OST0000 complete [10946.531264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10946.553264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10946.554994] Lustre: Skipped 1 previous similar message [10946.574070] Lustre: Skipped 1 previous similar message [10951.649364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10951.655186] Lustre: Skipped 1 previous similar message [10951.941848] Lustre: server umount lustre-MDT0000 complete [10956.139209] LustreError: 304845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781006922 with bad export cookie 2188773138584176954 [10956.141534] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10956.158293] LustreError: 304845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10956.486198] Lustre: server umount lustre-MDT0001 complete [10969.891406] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [10973.640549] Key type lgssc unregistered [10982.431367] LNet: 1 local NIs in recovery (showing 1): 192.168.203.135@tcp [10983.199889] LNet: 308624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10983.236914] LNetError: 308624:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11056.998037] LNet: Removed LNI 192.168.203.135@tcp [11058.502855] Key type .llcrypt unregistered [11058.505541] Key type ._llcrypt unregistered [11077.346992] Key type ._llcrypt registered [11077.351152] Key type .llcrypt registered [11077.434863] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [11092.370678] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [11093.316045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11093.415294] alg: No test for adler32 (adler32-zlib) [11094.518859] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [11094.709321] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [11096.423171] Key type lgssc registered [11097.851912] Lustre: Echo OBD driver; http://www.lustre.org/ [11130.294776] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 08:11:35 (1781007095) [11137.069833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11137.103118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11138.383383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11138.415893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11138.533500] Lustre: lustre-MDT0000: new disk, initializing [11138.604818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11138.620826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11143.141081] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11153.029718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11153.118925] Lustre: 311818:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11153.191811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11153.196171] Lustre: Skipped 1 previous similar message [11153.294788] Lustre: lustre-MDT0001: new disk, initializing [11153.406573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11153.446324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11153.465212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11157.555860] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11164.120754] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11167.633994] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11175.860549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11176.106733] Lustre: lustre-OST0000: new disk, initializing [11176.112923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11176.123491] Lustre: 313559:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11176.199797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11183.797620] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11184.207074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11184.231981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11184.320240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11191.894258] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11210.208631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11210.221131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11210.226161] LustreError: Skipped 1 previous similar message [11210.253220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11210.261770] Lustre: Skipped 1 previous similar message [11210.281132] Lustre: Skipped 1 previous similar message [11212.546811] Lustre: server umount lustre-OST0000 complete [11219.936244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11219.939472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11219.960418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11223.048601] Lustre: server umount lustre-MDT0000 complete [11227.036208] LustreError: 311811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781007193 with bad export cookie 10670761084112673607 [11227.043469] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11227.048707] LustreError: 311811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11227.498279] Lustre: server umount lustre-MDT0001 complete [11239.573538] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11242.226287] Key type lgssc unregistered [11242.506930] LNet: 315336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11242.515066] LNetError: 315336:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11242.543351] LNet: Removed LNI 192.168.203.135@tcp [11243.402444] Key type .llcrypt unregistered [11243.406475] Key type ._llcrypt unregistered [11262.021335] Key type ._llcrypt registered [11262.025683] Key type .llcrypt registered [11262.157781] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [11263.276699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11263.546616] alg: No test for adler32 (adler32-zlib) [11264.546552] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [11264.749241] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [11266.415253] Key type lgssc registered [11267.461996] Lustre: Echo OBD driver; http://www.lustre.org/ [11274.888973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11274.900464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11276.337799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11279.835315] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11286.497385] LustreError: 316590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11286.511794] LustreError: 316590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11287.652052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11288.040507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11293.297628] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11299.419220] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11303.577885] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11311.703367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11312.113661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11319.505023] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11326.356261] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11336.821444] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11338.419356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11344.353228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11344.378592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11344.409780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11346.912290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11346.918174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11346.928697] Lustre: Skipped 2 previous similar messages [11346.956815] Lustre: Skipped 1 previous similar message [11349.705570] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping) [11349.717532] Lustre: Skipped 1 previous similar message [11350.378690] Lustre: server umount lustre-MDT0000 complete [11352.043721] LustreError: 316590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11352.076949] LustreError: 316590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11354.630345] LustreError: 316572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781007320 with bad export cookie 7457322256052417701 [11354.639389] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11354.643119] LustreError: 316572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11354.826443] LustreError: 318204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11354.865047] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.35@tcp (stopping) [11357.153428] LustreError: 316586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11357.154636] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11359.207131] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11359.213963] Lustre: Skipped 1 previous similar message [11361.020983] Lustre: server umount lustre-MDT0001 complete [11365.429839] Lustre: server umount lustre-OST0000 complete [11374.550213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11379.051221] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11381.978450] Lustre: 320318:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.135@tcp [11384.115856] Lustre: server umount MGS complete [11392.335506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11392.746089] LustreError: 320891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11392.763559] LustreError: 320891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11392.820525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11397.264657] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11403.237960] LustreError: 320892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11403.253503] LustreError: 320892:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11407.816915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11408.190075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11412.977120] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11422.193625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11422.508236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11427.181808] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11431.418501] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11432.677199] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11435.980387] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11443.736181] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [11449.315344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11449.322061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11449.329510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11449.332078] Lustre: Skipped 1 previous similar message [11453.408496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11454.751610] Lustre: server umount lustre-OST0000 complete [11459.554660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11463.655624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11463.670218] Lustre: Skipped 1 previous similar message [11468.774048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11468.779327] Lustre: Skipped 6 previous similar messages [11473.375260] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11473.872177] Lustre: server umount lustre-MDT0000 complete [11473.899253] LustreError: 320892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11473.946785] LustreError: 320892:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11478.295275] LustreError: 322315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781007444 with bad export cookie 7457322256052418541 [11478.302459] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11478.314070] LustreError: 322315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11478.660928] Lustre: server umount lustre-MDT0001 complete [11489.926550] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11493.133690] Key type lgssc unregistered [11493.484809] LNet: 324070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11493.494167] LNetError: 324070:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11493.521810] LNet: Removed LNI 192.168.203.135@tcp [11494.440530] Key type .llcrypt unregistered [11494.444892] Key type ._llcrypt unregistered [11517.152560] Key type ._llcrypt registered [11517.155300] Key type .llcrypt registered [11517.276835] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [11535.885164] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [11538.161855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11538.223800] alg: No test for adler32 (adler32-zlib) [11539.416719] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [11539.834102] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [11541.583459] Key type lgssc registered [11543.059833] Lustre: Echo OBD driver; http://www.lustre.org/ [11583.283800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11583.300613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11584.730422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11584.791447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11584.911524] Lustre: lustre-MDT0000: new disk, initializing [11585.028528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11585.060522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11589.748821] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11601.470671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11601.591318] Lustre: 327201:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11601.662115] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11601.672173] Lustre: Skipped 1 previous similar message [11601.732809] Lustre: lustre-MDT0001: new disk, initializing [11601.933813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11601.972157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11601.984242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11605.981809] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11615.003115] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11619.333689] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11626.213461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11626.464963] Lustre: lustre-OST0000: new disk, initializing [11626.468549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11626.474407] Lustre: 328942:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11626.601456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11632.416292] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11634.824312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11634.834571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11634.965882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11640.523294] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11648.574936] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11649.039056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11654.539875] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11654.872049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11660.778680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11660.783429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11660.790235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11660.798863] Lustre: Skipped 1 previous similar message [11660.803191] LustreError: Skipped 1 previous similar message [11663.719033] Lustre: server umount lustre-OST0000 complete [11668.448541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11668.460857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11668.482567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11668.492747] Lustre: Skipped 1 previous similar message [11671.009695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11671.012228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11671.039054] Lustre: Skipped 1 previous similar message [11673.661804] Lustre: server umount lustre-MDT0000 complete [11677.275888] LustreError: 327192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781007643 with bad export cookie 2182017657985534144 [11677.276613] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11677.286502] LustreError: 327192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11677.716631] Lustre: server umount lustre-MDT0001 complete [11685.608930] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 08:20:50 (1781007650) [11700.324428] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [11710.081386] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [11741.540079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11741.780569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11741.805128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11741.912918] Lustre: lustre-MDT0000: new disk, initializing [11741.985372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11742.017729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11748.064447] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11762.050337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11762.370672] Lustre: 333520:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11762.523567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11762.531028] Lustre: Skipped 1 previous similar message [11762.716851] Lustre: lustre-MDT0001: new disk, initializing [11762.946859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11763.016176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11763.033469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11769.356951] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11778.525247] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11782.901991] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11791.471598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11791.767710] Lustre: lustre-OST0000: new disk, initializing [11791.775171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11791.785917] Lustre: 335264:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11791.937050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11798.579447] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11799.139852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11799.149298] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11799.234082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11806.692650] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11845.609210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11845.621712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11845.629830] LustreError: Skipped 1 previous similar message [11845.634788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11845.663461] Lustre: Skipped 1 previous similar message [11845.703906] Lustre: Skipped 1 previous similar message [11850.723407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11850.732210] Lustre: Skipped 1 previous similar message [11855.327414] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11855.502737] Lustre: server umount lustre-OST0000 complete [11860.448513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11860.459848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11860.480698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11864.840434] Lustre: server umount lustre-MDT0000 complete [11866.083534] LustreError: 333534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.095122] LustreError: 333534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11868.255914] LustreError: 333513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781007834 with bad export cookie 2182017657985535026 [11868.264994] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11868.266339] LustreError: 333513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11868.615164] Lustre: server umount lustre-MDT0001 complete [11878.862710] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11881.465404] Key type lgssc unregistered [11881.792252] LNet: 337288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11881.817975] LNetError: 337288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11881.832807] LNet: Removed LNI 192.168.203.135@tcp [11882.588502] Key type .llcrypt unregistered [11882.593997] Key type ._llcrypt unregistered [11981.162875] Key type ._llcrypt registered [11981.165554] Key type .llcrypt registered [11981.259500] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11981.619626] Key type .llcrypt unregistered [11981.623879] Key type ._llcrypt unregistered [12067.914973] Key type ._llcrypt registered [12067.921449] Key type .llcrypt registered [12068.124347] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12068.426618] Key type .llcrypt unregistered [12068.428932] Key type ._llcrypt unregistered [12096.869387] Key type ._llcrypt registered [12096.872304] Key type .llcrypt registered [12096.962709] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12097.251265] Key type .llcrypt unregistered [12097.259641] Key type ._llcrypt unregistered [12124.163518] Key type ._llcrypt registered [12124.166817] Key type .llcrypt registered [12124.319354] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12124.599426] Key type .llcrypt unregistered [12124.602227] Key type ._llcrypt unregistered [12192.797725] Key type ._llcrypt registered [12192.801189] Key type .llcrypt registered [12192.921855] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12193.192306] Key type .llcrypt unregistered [12193.194537] Key type ._llcrypt unregistered [12217.726797] Key type ._llcrypt registered [12217.728846] Key type .llcrypt registered [12217.810735] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12218.062551] Key type .llcrypt unregistered [12218.067255] Key type ._llcrypt unregistered [12243.402877] Key type ._llcrypt registered [12243.407672] Key type .llcrypt registered [12243.500848] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12243.756592] Key type .llcrypt unregistered [12243.760920] Key type ._llcrypt unregistered [12268.016293] Key type ._llcrypt registered [12268.018180] Key type .llcrypt registered [12268.098340] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12268.303396] Key type .llcrypt unregistered [12268.307860] Key type ._llcrypt unregistered [12292.961515] Key type ._llcrypt registered [12292.964479] Key type .llcrypt registered [12293.074235] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12293.329220] Key type .llcrypt unregistered [12293.335558] Key type ._llcrypt unregistered [12319.072355] Key type ._llcrypt registered [12319.076438] Key type .llcrypt registered [12319.191136] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12319.408522] Key type .llcrypt unregistered [12319.412285] Key type ._llcrypt unregistered [12362.120711] Key type ._llcrypt registered [12362.122678] Key type .llcrypt registered [12362.192361] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12362.383881] Key type .llcrypt unregistered [12362.387187] Key type ._llcrypt unregistered [12381.916880] Key type ._llcrypt registered [12381.922540] Key type .llcrypt registered [12381.985420] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12382.187322] Key type .llcrypt unregistered [12382.190426] Key type ._llcrypt unregistered [12402.277671] Key type ._llcrypt registered [12402.279583] Key type .llcrypt registered [12402.346812] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12402.563414] Key type .llcrypt unregistered [12402.567629] Key type ._llcrypt unregistered [12438.702554] Key type ._llcrypt registered [12438.706386] Key type .llcrypt registered [12438.786753] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [12449.186717] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [12450.056206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12450.092309] alg: No test for adler32 (adler32-zlib) [12451.108374] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [12451.268100] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [12452.911201] Key type lgssc registered [12453.752744] Lustre: Echo OBD driver; http://www.lustre.org/ [12478.489879] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12478.505800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12479.731660] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12479.756840] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12479.810348] Lustre: tst250fs-MDT0000: new disk, initializing [12479.860574] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12479.872244] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12482.422189] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12489.723994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12489.822963] Lustre: 357108:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12489.881483] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12489.886093] Lustre: Skipped 1 previous similar message [12489.965954] Lustre: tst250fs-MDT0001: new disk, initializing [12490.031574] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12490.069966] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12490.079801] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12492.570913] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12495.880391] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12500.121253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12500.305633] Lustre: tst250fs-OST0000: new disk, initializing [12500.309181] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12500.318675] Lustre: 358704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12500.398617] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12504.265180] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12504.342325] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12504.357185] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12504.447223] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12510.838184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12510.945743] Lustre: tst250fs-OST0001: new disk, initializing [12510.950288] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12510.958207] Lustre: 359559:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12511.031583] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12512.868072] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12512.874183] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12512.922775] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12514.902953] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12518.612910] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12533.734192] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12533.738205] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12533.741369] Lustre: Skipped 3 previous similar messages [12533.750950] Lustre: Skipped 3 previous similar messages [12538.852036] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12538.855013] Lustre: Skipped 1 previous similar message [12539.315774] Lustre: server umount tst250fs-MDT0000 complete [12541.920565] LustreError: 357102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781008508 with bad export cookie 13433983517191544989 [12541.927330] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12541.929325] LustreError: 357102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12542.162346] Lustre: server umount tst250fs-MDT0001 complete [12553.910672] Lustre: server umount tst250fs-OST0000 complete [12566.846043] Lustre: server umount tst250fs-OST0001 complete [12577.003363] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [12581.264159] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [12601.890887] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:36:06 (1781008566) [12603.302508] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12624.887577] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [12640.134495] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [12683.104424] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12364 sec ========= 08:37:27 (1781008647) [12686.318582] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:37:29 (1781008649) === [12690.669425] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:37:35 (1781008655) === [12733.109597] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12736.268200] Key type lgssc unregistered [12736.551816] LNet: 368614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12736.556795] LNetError: 368614:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12736.594525] LNet: Removed LNI 192.168.203.135@tcp [12737.525610] Key type .llcrypt unregistered [12737.527502] Key type ._llcrypt unregistered