[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 471347470 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003213] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007741] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008041] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.011149] LSM: Security Framework initializing [ 0.012068] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.015068] *** VALIDATE selinux *** [ 0.024168] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029423] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032106] *** VALIDATE tmpfs *** [ 0.033443] *** VALIDATE proc *** [ 0.034245] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036249] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043089] debug: unmapping init [mem 0xffffffffb8c59000-0xffffffffb8c60fff] [ 0.045971] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046705] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047015] ... version: 2 [ 0.047933] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052008] ... event mask: 000000070000000f [ 0.053296] rcu: Hierarchical SRCU implementation. [ 0.055526] smp: Bringing up secondary CPUs ... [ 0.056590] x86: Booting SMP configuration: [ 0.057035] .... node #0, CPUs: #1 #2 #3 [ 0.060270] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140421] node 0 deferred pages initialised in 75ms [ 0.143538] devtmpfs: initialized [ 0.145177] x86/mm: Memory block size: 128MB [ 0.148284] gcov: version magic: 0x41383552 [ 0.150367] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152526] pinctrl core: initialized pinctrl subsystem [ 0.153297] [ 0.153887] ************************************************************* [ 0.154021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155018] ** ** [ 0.156011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157013] ** ** [ 0.158013] ** This means that this kernel is built to expose internal ** [ 0.159013] ** IOMMU data structures, which may compromise security on ** [ 0.160012] ** your system. ** [ 0.161012] ** ** [ 0.162028] ** If you see this message and you are not debugging the ** [ 0.163028] ** kernel, report this immediately to your vendor! ** [ 0.164013] ** ** [ 0.165014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166012] ************************************************************* [ 0.167686] NET: Registered protocol family 16 [ 0.168457] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.171483] cpuidle: using governor menu [ 0.174012] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176504] PCI: Using configuration type 1 for base access [ 0.179097] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187210] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190103] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.194052] cryptd: max_cpu_qlen set to 1000 [ 0.197258] ACPI: Added _OSI(Module Device) [ 0.198012] ACPI: Added _OSI(Processor Device) [ 0.200010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.201008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.205786] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.210457] ACPI: Interpreter enabled [ 0.211038] ACPI: PM: (supports S0 S3 S4 S5) [ 0.212006] ACPI: Using IOAPIC for interrupt routing [ 0.213074] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.215286] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223611] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.225030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.230055] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.233145] acpiphp: Slot [2] registered [ 0.234070] acpiphp: Slot [5] registered [ 0.234932] acpiphp: Slot [6] registered [ 0.236086] acpiphp: Slot [7] registered [ 0.237127] acpiphp: Slot [8] registered [ 0.239116] acpiphp: Slot [9] registered [ 0.240130] acpiphp: Slot [10] registered [ 0.242107] acpiphp: Slot [3] registered [ 0.243114] acpiphp: Slot [4] registered [ 0.245086] acpiphp: Slot [11] registered [ 0.246091] acpiphp: Slot [12] registered [ 0.248198] acpiphp: Slot [13] registered [ 0.250088] acpiphp: Slot [14] registered [ 0.251130] acpiphp: Slot [15] registered [ 0.253097] acpiphp: Slot [16] registered [ 0.254184] acpiphp: Slot [17] registered [ 0.256173] acpiphp: Slot [18] registered [ 0.257126] acpiphp: Slot [19] registered [ 0.259160] acpiphp: Slot [20] registered [ 0.260101] acpiphp: Slot [21] registered [ 0.262101] acpiphp: Slot [22] registered [ 0.264124] acpiphp: Slot [23] registered [ 0.265085] acpiphp: Slot [24] registered [ 0.266170] acpiphp: Slot [25] registered [ 0.268091] acpiphp: Slot [26] registered [ 0.270109] acpiphp: Slot [27] registered [ 0.271154] acpiphp: Slot [28] registered [ 0.273089] acpiphp: Slot [29] registered [ 0.274132] acpiphp: Slot [30] registered [ 0.276181] acpiphp: Slot [31] registered [ 0.277097] PCI host bridge to bus 0000:00 [ 0.279028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.281022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.284065] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.286025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.289026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.292104] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.294166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.297070] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.300293] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.310089] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.315613] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.318017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.320016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.323026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.326342] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.327639] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.329035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.331605] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.335013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.350014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.357018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.361151] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.377028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.386018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.412016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.425673] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.436073] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.444019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.459022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.471708] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.480022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.491016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.508017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.519586] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.532015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.542016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.567014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.579280] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.584018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.590015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.607022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.617052] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.625016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.632016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.649020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.659305] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.664281] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.666234] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.670300] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.672226] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.676190] iommu: Default domain type: Passthrough [ 0.678482] SCSI subsystem initialized [ 0.679072] ACPI: bus type USB registered [ 0.679936] usbcore: registered new interface driver usbfs [ 0.681055] usbcore: registered new interface driver hub [ 0.682059] usbcore: registered new device driver usb [ 0.684247] pps_core: LinuxPPS API ver. 1 registered [ 0.686009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.689055] PTP clock support registered [ 0.691094] EDAC MC: Ver: 3.0.0 [ 0.693115] PCI: Using ACPI for IRQ routing [ 0.694395] NetLabel: Initializing [ 0.696012] NetLabel: domain hash size = 128 [ 0.697007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.699177] NetLabel: unlabeled traffic allowed by default [ 0.702115] vgaarb: loaded [ 0.703253] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.704010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.713971] clocksource: Switched to clocksource kvm-clock [ 0.870537] VFS: Disk quotas dquot_6.6.0 [ 0.872340] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.875094] *** VALIDATE ramfs *** [ 0.876696] *** VALIDATE hugetlbfs *** [ 0.878319] pnp: PnP ACPI init [ 0.881099] pnp: PnP ACPI: found 6 devices [ 0.897490] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.899628] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.900949] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.902411] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.903761] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.905883] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.909209] NET: Registered protocol family 2 [ 0.911202] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.915558] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.917986] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.921978] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.924500] TCP: Hash tables configured (established 65536 bind 65536) [ 0.926382] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.929195] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.930946] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.932670] NET: Registered protocol family 1 [ 0.934531] RPC: Registered named UNIX socket transport module. [ 0.935663] RPC: Registered udp transport module. [ 0.936684] RPC: Registered tcp transport module. [ 0.937712] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.939064] NET: Registered protocol family 44 [ 0.939961] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.941437] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.942638] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.943912] PCI: CLS 0 bytes, default 64 [ 0.945548] Unpacking initramfs... [ 2.314809] debug: unmapping init [mem 0xffff9d3a3cc54000-0xffff9d3a3ffbffff] [ 2.318719] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.320911] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.322857] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.881482] Initialise system trusted keyrings [ 2.882736] Key type blacklist registered [ 2.884133] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.894212] zbud: loaded [ 2.897221] *** VALIDATE nfs *** [ 2.898527] *** VALIDATE nfs4 *** [ 2.900037] pstore: using deflate compression [ 2.903915] Platform Keyring initialized [ 3.018493] NET: Registered protocol family 38 [ 3.021343] Key type asymmetric registered [ 3.022856] Asymmetric key parser 'x509' registered [ 3.024709] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.027696] io scheduler mq-deadline registered [ 3.029062] io scheduler kyber registered [ 3.030768] io scheduler bfq registered [ 3.032627] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.035634] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.038541] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.041444] ACPI: Power Button [PWRF] [ 3.047818] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.055662] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.066502] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.072826] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.088982] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.120518] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.149689] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.153294] Non-volatile memory driver v1.3 [ 3.154895] Linux agpgart interface v0.103 [ 3.191988] virtio_blk virtio1: [vda] 146960 512-byte logical blocks (75.2 MB/71.8 MiB) [ 3.196535] vda: detected capacity change from 0 to 75243520 [ 3.208693] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.210804] vdb: detected capacity change from 0 to 1073741824 [ 3.224843] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.227936] vdc: detected capacity change from 0 to 2621440000 [ 3.247484] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.250532] vdd: detected capacity change from 0 to 2621440000 [ 3.266475] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.269536] vde: detected capacity change from 0 to 4294967296 [ 3.285127] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.288061] vdf: detected capacity change from 0 to 4294967296 [ 3.295553] libphy: Fixed MDIO Bus: probed [ 3.305096] usbcore: registered new interface driver usbserial_generic [ 3.307012] usbserial: USB Serial support registered for generic [ 3.308983] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.313423] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.315017] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.317226] mousedev: PS/2 mouse device common for all mice [ 3.324511] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.326682] rtc_cmos 00:05: RTC can wake from S4 [ 3.330080] rtc_cmos 00:05: registered as rtc0 [ 3.331431] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.332290] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.337955] intel_pstate: CPU model not supported [ 3.341218] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.345702] hid: raw HID events driver (C) Jiri Kosina [ 3.347222] usbcore: registered new interface driver usbhid [ 3.348689] usbhid: USB HID core driver [ 3.349877] drop_monitor: Initializing network drop monitor service [ 3.351600] Initializing XFRM netlink socket [ 3.353154] NET: Registered protocol family 10 [ 3.355376] Segment Routing with IPv6 [ 3.356826] NET: Registered protocol family 17 [ 3.359614] mpls_gso: MPLS GSO support [ 3.367137] RAS: Correctable Errors collector initialized. [ 3.368628] AVX version of gcm_enc/dec engaged. [ 3.369679] AES CTR mode by8 optimization enabled [ 3.438777] sched_clock: Marking stable (3438755439, 0)->(4318864098, -880108659) [ 3.441500] registered taskstats version 1 [ 3.444257] Loading compiled-in X.509 certificates [ 3.446912] zswap: loaded using pool lzo/zbud [ 3.472522] Key type big_key registered [ 3.485627] Key type encrypted registered [ 3.487166] ima: No TPM chip found, activating TPM-bypass! [ 3.489374] ima: Allocated hash algorithm: sha1 [ 3.490983] ima: No architecture policies found [ 3.492782] evm: Initialising EVM extended attributes: [ 3.494678] evm: security.selinux [ 3.495903] evm: security.ima [ 3.497131] evm: security.capability [ 3.498486] evm: HMAC attrs: 0x1 [ 3.500934] rtc_cmos 00:05: setting system clock to 2026-08-27 16:51:36 UTC (1787849496) [ 3.509134] debug: unmapping init [mem 0xffffffffb9c03000-0xffffffffb9dfffff] [ 3.511784] debug: unmapping init [mem 0xffffffffb8982000-0xffffffffb8c58fff] [ 3.518267] Write protecting the kernel read-only data: 28672k [ 3.522145] debug: unmapping init [mem 0xffffffffb7003000-0xffffffffb71fffff] [ 3.524395] debug: unmapping init [mem 0xffffffffb7914000-0xffffffffb79fffff] [ 3.573308] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.580425] systemd[1]: Detected virtualization kvm. [ 3.581777] systemd[1]: Detected architecture x86-64. [ 3.583012] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.608267] systemd[1]: No hostname configured. [ 3.609933] systemd[1]: Set hostname to . [ 3.611715] random: systemd: uninitialized urandom read (16 bytes read) [ 3.613851] systemd[1]: Initializing machine ID from random generator. [ 3.778468] random: systemd: uninitialized urandom read (16 bytes read) [ 3.781305] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.787173] random: systemd: uninitialized urandom read (16 bytes read) [ 3.790967] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.797607] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.896633] device-mapper: uevent: version 1.0.3 [ 4.899863] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.032244] virtio_net virtio0 ens2: renamed from eth0 [ 6.044707] random: fast init done [ 6.367951] scsi host0: ata_piix [ 6.479728] scsi host1: ata_piix [ 6.481254] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.483840] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.396268] random: crng init done [ 12.399879] random: 7 urandom warning(s) missed due to ratelimiting [ 14.179538] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.151600] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.239810] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.628867] SELinux: Disabled at runtime. [ 17.697744] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.706166] systemd[1]: Detected virtualization kvm. [ 17.708308] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.547290] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.551559] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.559413] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.564400] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.568480] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.579415] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.588900] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ 20.606191] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 21.745093] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.566235] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.622796] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.821587] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.911707] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit)[ 28.099543] Key type dns_resolver registered [ 28.463856] NFS: Registering the id_resolver key type [ 28.466428] Key type id_resolver registered [ 28.468363] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg158-server login: [ 58.437025] hrtimer: interrupt took 8014633 ns [ 87.849480] libcfs: loading out-of-tree module taints kernel. [ 87.874380] Key type ._llcrypt registered [ 87.877296] Key type .llcrypt registered [ 87.946969] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 105.838913] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 107.653383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 107.659613] alg: No test for adler32 (adler32-zlib) [ 108.934750] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 109.724654] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 111.399213] Key type lgssc registered [ 113.030283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.791048] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.620728] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 187.790912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 187.848488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.218614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.292199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 189.425806] Lustre: lustre-MDT0000: new disk, initializing [ 189.540603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.580560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.513384] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.100438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.208387] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 209.254540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 209.257614] Lustre: Skipped 1 previous similar message [ 209.352289] Lustre: lustre-MDT0001: new disk, initializing [ 209.406337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.442490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 209.450148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 214.086888] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.378387] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 230.248441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 230.797899] Lustre: lustre-OST0000: new disk, initializing [ 230.802134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 230.810582] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 230.914821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.640031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 237.657517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 237.737065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 238.148443] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.722621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.921113] Lustre: lustre-OST0001: new disk, initializing [ 254.931376] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 254.952623] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 255.127992] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 261.829503] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.762208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 262.778692] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 262.866706] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 274.152684] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 281.422125] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 287.947660] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing check_logdir /tmp/testlogs/ [ 293.525408] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing yml_node [ 298.974288] Lustre: DEBUG MARKER: Client: 2.17.57.98 [ 301.594469] Lustre: DEBUG MARKER: MDS: 2.17.57.98 [ 304.345314] Lustre: DEBUG MARKER: OSS: 2.17.57.98 [ 306.056756] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 12:56:37 EDT 2026 [ 324.265197] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 326.051486] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 342.497961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 342.510307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.530788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.056162] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.060295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.074283] Lustre: Skipped 2 previous similar messages [ 345.097373] Lustre: Skipped 3 previous similar messages [ 350.446775] Lustre: server umount lustre-MDT0000 complete [ 354.463940] LustreError: 10336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849847 with bad export cookie 1085272885383649212 [ 354.467992] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 354.483234] LustreError: 10336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 355.296488] LustreError: 8700:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 355.302308] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.351130] LustreError: 8700:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 355.409261] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.331456] LustreError: 6517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.331600] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.371747] Lustre: Skipped 2 previous similar messages [ 361.002974] Lustre: server umount lustre-MDT0001 complete [ 366.077434] Lustre: server umount lustre-OST0000 complete [ 371.997622] Lustre: server umount lustre-OST0001 complete [ 384.056602] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 405.706661] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 448.566225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.767180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 448.786982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 448.848710] Lustre: lustre-MDT0000: new disk, initializing [ 448.926776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.940969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 454.071884] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 464.430714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 464.572163] Lustre: 15098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 464.625238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 464.629506] Lustre: Skipped 1 previous similar message [ 464.724823] Lustre: lustre-MDT0001: new disk, initializing [ 464.853647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 464.907425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 464.925947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 470.131208] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 479.985734] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 486.671508] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 494.613397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 494.973624] Lustre: lustre-OST0000: new disk, initializing [ 494.978980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 494.990618] Lustre: 17064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 495.095481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.353486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 496.367375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 496.528828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 501.352970] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 511.333446] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 516.798410] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 517.150295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 522.239293] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 522.434968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 527.339285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.340455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 527.355653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 527.364094] Lustre: Skipped 2 previous similar messages [ 530.130354] Lustre: server umount lustre-OST0000 complete [ 536.545924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 536.559942] LustreError: Skipped 1 previous similar message [ 536.565817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.579789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 536.585644] Lustre: Skipped 1 previous similar message [ 539.624674] Lustre: server umount lustre-MDT0000 complete [ 542.688603] LustreError: 15109:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.714892] LustreError: 15109:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 543.829970] LustreError: 15089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850036 with bad export cookie 1085272885383650745 [ 543.836663] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.169678] Lustre: server umount lustre-MDT0001 complete [ 551.284674] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:00:42 (1787850042) [ 572.270527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.987470] LustreError: 19952:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.114392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 578.033080] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 578.532445] LustreError: 19953:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.561428] LustreError: 19953:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 588.138826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.988731] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 602.772091] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 610.291110] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 612.923094] Lustre: Failing over lustre-MDT0000 [ 613.147430] Lustre: server umount lustre-MDT0000 complete [ 614.374228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 614.374400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.391532] Lustre: Skipped 2 previous similar messages [ 614.395609] LustreError: 19949:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 614.412146] LustreError: 19949:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 630.751317] Lustre: 3648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850107/real 1787850107] req@ffff9d3983ec5680 x1874696184429056/t0(0) o400->MGC192.168.201.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787850123 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 630.766910] LustreError: 19952:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.789677] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 630.832265] LustreError: 19952:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 634.236638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.389196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 641.398517] Lustre: Skipped 1 previous similar message [ 641.463505] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 646.168689] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 646.628402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 646.632989] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 646.672406] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 655.665255] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 661.927446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.939806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 661.941753] Lustre: Skipped 1 previous similar message [ 661.950923] Lustre: Skipped 3 previous similar messages [ 674.271776] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 674.440210] Lustre: server umount lustre-MDT0000 complete [ 677.353046] LustreError: 19948:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.378755] LustreError: 19948:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 678.235913] LustreError: 19933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850171 with bad export cookie 1085272885383652166 [ 678.237877] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 678.252743] LustreError: 19933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 678.785442] Lustre: server umount lustre-MDT0001 complete [ 688.838137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.353335] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 704.155143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.106212] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 719.707749] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 726.965874] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 729.582139] Lustre: Failing over lustre-MDT0000 [ 729.887325] Lustre: server umount lustre-MDT0000 complete [ 730.082942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 730.092289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 744.936227] LustreError: 23295:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 744.959517] LustreError: 23295:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 750.566774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 750.656703] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 750.934305] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 756.194673] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 756.197112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 756.220314] Lustre: Skipped 1 previous similar message [ 756.250316] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 756.525745] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 768.056824] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 774.955285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 774.962533] Lustre: Skipped 4 previous similar messages [ 779.202407] Lustre: server umount lustre-MDT0000 complete [ 783.697509] LustreError: 24864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850276 with bad export cookie 1085272885383653104 [ 783.706326] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 783.718489] LustreError: 24864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 784.112674] Lustre: server umount lustre-MDT0001 complete [ 794.460991] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:04:46 (1787850286) [ 813.837557] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 826.531875] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 877.044303] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 890.458546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.653942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 890.675244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 890.728474] Lustre: lustre-MDT0000: new disk, initializing [ 890.777356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.781247] Lustre: Skipped 3 previous similar messages [ 890.801208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 896.441982] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 907.928921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.044322] Lustre: 29671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 908.263612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 913.503757] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 924.096979] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 931.346493] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 938.513258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 938.921584] Lustre: 31642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 940.555318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 940.614300] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 940.684274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 940.697413] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 945.863440] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 955.799468] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 962.660425] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 986.601340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 986.612063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.622188] Lustre: Skipped 3 previous similar messages [ 986.634199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 996.319119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 996.776835] Lustre: server umount lustre-OST0000 complete [ 1015.264153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1015.491224] Lustre: server umount lustre-MDT0000 complete [ 1017.834175] LustreError: 29679:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1017.860598] LustreError: 29679:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 1019.622775] LustreError: 29663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850512 with bad export cookie 1085272885383653510 [ 1019.637761] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1019.644691] LustreError: 29663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1020.155937] Lustre: server umount lustre-MDT0001 complete [ 1033.262856] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1036.747972] Key type lgssc unregistered [ 1037.182692] LNet: 33627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.205977] LNetError: 33627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.245700] LNet: Removed LNI 192.168.201.158@tcp [ 1038.428157] Key type .llcrypt unregistered [ 1038.435593] Key type ._llcrypt unregistered [ 1068.044141] Key type ._llcrypt registered [ 1068.046438] Key type .llcrypt registered [ 1068.225635] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1069.185423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1069.240916] alg: No test for adler32 (adler32-zlib) [ 1070.319642] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 1070.577789] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1072.255344] Key type lgssc registered [ 1073.311871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1082.598924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1082.611184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.099744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1089.438739] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1094.633760] LustreError: 35042:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.667396] LustreError: 35042:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1099.200423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.553198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1105.569972] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1113.129240] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1119.298326] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1126.455670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1126.815274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.966476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1133.704684] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1142.628490] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1144.313093] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1159.569324] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:10:51 (1787850651) [ 1161.617101] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1163.895750] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1171.692940] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:11:03 (1787850663) [ 1180.695219] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:11:12 (1787850672) [ 1181.807649] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1191.367447] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1191.372646] Lustre: Skipped 2 previous similar messages [ 1193.451553] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:11:25 (1787850685) [ 1195.933072] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1195.947212] Lustre: Skipped 2 previous similar messages [ 1204.697924] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1204.704921] Lustre: Skipped 4 previous similar messages [ 1220.562676] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:11:51 (1787850711) [ 1223.884205] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1227.973136] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1237.414682] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:12:08 (1787850728) [ 1239.963905] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1239.966692] Lustre: Skipped 6 previous similar messages [ 1257.750433] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:12:28 (1787850748) [ 1277.437123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.446576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1277.473524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1277.523541] Lustre: Skipped 1 previous similar message [ 1282.530368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1282.537177] Lustre: Skipped 1 previous similar message [ 1287.658044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1287.667281] Lustre: Skipped 1 previous similar message [ 1290.211609] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1290.455699] Lustre: server umount lustre-OST0000 complete [ 1297.891150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.901975] Lustre: Skipped 1 previous similar message [ 1297.905711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1302.608965] Lustre: server umount lustre-MDT0000 complete [ 1303.024216] LustreError: 35037:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.059390] LustreError: 35037:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1307.497021] LustreError: 38603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850800 with bad export cookie 806221000614879845 [ 1307.500901] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1307.507883] LustreError: 38603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1307.702516] Lustre: server umount lustre-MDT0001 complete [ 1320.405554] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1323.349867] Key type lgssc unregistered [ 1323.771426] LNet: 42913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.785199] LNetError: 42913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.820230] LNet: Removed LNI 192.168.201.158@tcp [ 1324.660247] Key type .llcrypt unregistered [ 1324.662367] Key type ._llcrypt unregistered [ 1351.255678] Key type ._llcrypt registered [ 1351.258364] Key type .llcrypt registered [ 1351.371911] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 1370.212291] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1371.940431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1371.959920] alg: No test for adler32 (adler32-zlib) [ 1373.544475] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 1374.137897] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1375.863382] Key type lgssc registered [ 1377.728207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1425.606987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1425.638548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1426.968490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1427.021666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1427.077965] Lustre: lustre-MDT0000: new disk, initializing [ 1427.197314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.231841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1433.525667] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1443.526686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.623888] Lustre: 46080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1443.679546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1443.682459] Lustre: Skipped 1 previous similar message [ 1443.778956] Lustre: lustre-MDT0001: new disk, initializing [ 1443.903179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1443.971874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1443.979829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1448.621813] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1458.426972] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1464.943319] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1472.021711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1472.247479] Lustre: lustre-OST0000: new disk, initializing [ 1472.250459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1472.257728] Lustre: 48056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1472.346352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.693695] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1481.887970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1481.905936] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1482.086533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1489.497338] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1494.159632] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1494.435725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1499.428447] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1499.703153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1502.688364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1502.692864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1502.705888] Lustre: Skipped 1 previous similar message [ 1502.706379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1502.717586] Lustre: Skipped 1 previous similar message [ 1507.813228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1507.825047] Lustre: Skipped 1 previous similar message [ 1508.509139] Lustre: server umount lustre-OST0000 complete [ 1512.930862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.939759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1512.955543] Lustre: Skipped 1 previous similar message [ 1512.976520] Lustre: Skipped 1 previous similar message [ 1518.051934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1518.057259] Lustre: Skipped 1 previous similar message [ 1521.015579] Lustre: server umount lustre-MDT0000 complete [ 1524.998414] LustreError: 46073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851017 with bad export cookie 12494200818312808261 [ 1525.005704] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1525.011411] LustreError: 46073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1525.570253] Lustre: server umount lustre-MDT0001 complete [ 1535.574433] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:17:06 (1787851026) [ 1543.387714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.072694] LustreError: 50519:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1544.092929] LustreError: 50519:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1544.197904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1548.823482] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1549.285700] LustreError: 50520:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1554.401900] LustreError: 50519:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1559.533183] LustreError: 50520:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1560.508183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.887771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1566.350771] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1575.839207] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1583.060891] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1591.628356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1592.074965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.951154] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1606.073828] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1620.881637] LustreError: 53067:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1621.899629] Lustre: Modifying parameter general.timeout=1 in log params [ 1623.917374] Lustre: Modifying parameter general.timeout=3 in log params [ 1623.927540] Lustre: Skipped 1 previous similar message [ 1626.310842] Lustre: Modifying parameter general.timeout=5 in log params [ 1626.313085] Lustre: Skipped 1 previous similar message [ 1631.267335] Lustre: Modifying parameter general.timeout=9 in log params [ 1631.277857] Lustre: Skipped 3 previous similar messages [ 1640.171937] Lustre: Modifying parameter general.timeout=17 in log params [ 1640.176810] Lustre: Skipped 7 previous similar messages [ 1656.839939] Lustre: Modifying parameter general.timeout=31 in log params [ 1656.842479] Lustre: Skipped 13 previous similar messages [ 1689.522913] Lustre: Modifying parameter general.timeout=61 in log params [ 1689.526851] Lustre: Skipped 29 previous similar messages [ 1753.537996] Lustre: Modifying parameter general.timeout=124 in log params [ 1753.541661] Lustre: Skipped 62 previous similar messages [ 1770.402945] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:21:02 (1787851262) [ 1774.089101] Lustre: 60029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1775.879515] Lustre: 60127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1778.050744] Lustre: 60225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1782.590842] Lustre: 60438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1782.618798] Lustre: 60438:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1820.545786] Lustre: 61901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1820.573908] Lustre: 61901:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1820.629437] LustreError: 61901:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1828.211982] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:21:59 (1787851319) [ 1845.735986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1845.743557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1845.762753] Lustre: Skipped 2 previous similar messages [ 1845.781474] Lustre: Skipped 2 previous similar messages [ 1851.981689] Lustre: server umount lustre-MDT0000 complete [ 1855.977760] LustreError: 50515:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1856.016236] LustreError: 50515:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1856.630212] LustreError: 50499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851349 with bad export cookie 12494200818312809143 [ 1856.636345] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1856.660879] LustreError: 50499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1857.053293] Lustre: server umount lustre-MDT0001 complete [ 1871.814989] Lustre: server umount lustre-OST0000 complete [ 1901.087813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.403690] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1901.457776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1901.470714] Lustre: Skipped 28 previous similar messages [ 1901.856987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.217546] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1915.845522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.006054] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1916.042157] Lustre: 64436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1922.318890] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1930.791582] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1938.473685] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1947.013035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1947.318523] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1947.618412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.632198] Lustre: Skipped 1 previous similar message [ 1954.416392] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1962.588810] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1964.508664] Lustre: 67006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1977.830981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.837723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1977.851963] Lustre: Skipped 1 previous similar message [ 1977.852322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1977.865199] Lustre: Skipped 4 previous similar messages [ 1979.752616] Lustre: server umount lustre-OST0000 complete [ 1988.065622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1988.072696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.087046] Lustre: Skipped 1 previous similar message [ 1990.546527] Lustre: server umount lustre-MDT0000 complete [ 1993.188844] LustreError: 64444:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.218348] LustreError: 64444:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1995.071283] LustreError: 64429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851488 with bad export cookie 12494200818312812342 [ 1995.086111] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1995.089169] LustreError: 64429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1995.995783] Lustre: server umount lustre-MDT0001 complete [ 2010.704403] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2014.928147] Key type lgssc unregistered [ 2015.316607] LNet: 68219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.325602] LNetError: 68219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.348793] LNet: Removed LNI 192.168.201.158@tcp [ 2016.783483] Key type .llcrypt unregistered [ 2016.786801] Key type ._llcrypt unregistered [ 2048.234320] Key type ._llcrypt registered [ 2048.236201] Key type .llcrypt registered [ 2048.410160] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2049.750462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2049.880894] alg: No test for adler32 (adler32-zlib) [ 2051.016578] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2051.341211] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2053.063455] Key type lgssc registered [ 2055.025294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2064.039204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2064.077736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.777325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.175407] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2075.107187] LustreError: 69636:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.124426] LustreError: 69636:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2080.226235] LustreError: 69635:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.981819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.720846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2087.642459] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2097.330915] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2104.769395] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2112.877179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2113.756073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.837229] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2131.687334] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2144.979073] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2156.002033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.011678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2156.018613] Lustre: Skipped 2 previous similar messages [ 2156.036520] Lustre: Skipped 2 previous similar messages [ 2161.124941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2161.144133] Lustre: Skipped 2 previous similar messages [ 2161.905806] Lustre: server umount lustre-MDT0000 complete [ 2165.949401] LustreError: 69615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851658 with bad export cookie 1034764231638002186 [ 2165.960382] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2166.243475] LustreError: 69632:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.246856] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2166.409451] Lustre: server umount lustre-MDT0001 complete [ 2172.907254] Lustre: server umount lustre-OST0000 complete [ 2206.417487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.817645] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2206.856980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2207.161249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2212.844549] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2225.346288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2225.697369] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2225.776078] Lustre: 74508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2226.415747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2232.549600] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2242.418382] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2250.489356] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2260.182254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.450774] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2261.085613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.230603] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2280.151743] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2282.097801] Lustre: 77085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2285.127319] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2290.660192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2290.662269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.674929] LustreError: Skipped 1 previous similar message [ 2290.676780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2290.711654] Lustre: Skipped 1 previous similar message [ 2295.781482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2295.785861] Lustre: Skipped 2 previous similar messages [ 2296.635817] Lustre: server umount lustre-OST0000 complete [ 2302.950990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2302.972323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.011389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2308.285479] Lustre: server umount lustre-MDT0000 complete [ 2311.139461] LustreError: 74519:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2311.180553] LustreError: 74519:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2314.136066] LustreError: 74501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851807 with bad export cookie 1034764231638003040 [ 2314.156625] LustreError: 74501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2314.162498] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.073161] Lustre: server umount lustre-MDT0001 complete [ 2335.576830] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2339.536554] Key type lgssc unregistered [ 2339.902664] LNet: 78322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.910263] LNetError: 78322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.960554] LNet: Removed LNI 192.168.201.158@tcp [ 2340.890967] Key type .llcrypt unregistered [ 2340.893074] Key type ._llcrypt unregistered [ 2368.714785] Key type ._llcrypt registered [ 2368.717367] Key type .llcrypt registered [ 2368.893090] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2370.558956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2370.580977] alg: No test for adler32 (adler32-zlib) [ 2371.687663] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2372.028310] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2373.687510] Key type lgssc registered [ 2375.338096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2384.376351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2384.397635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.024244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2391.484345] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2395.121447] LustreError: 79741:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.151688] LustreError: 79741:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2400.224719] LustreError: 79740:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2401.274866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.004651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2406.893215] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2415.875363] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2422.554288] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2429.616820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2430.219021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.715348] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2443.627843] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2453.693801] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2454.515603] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2456.139609] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2456.146189] Lustre: Skipped 1 previous similar message [ 2458.734865] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2458.738865] Lustre: Skipped 2 previous similar messages [ 2463.077472] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2463.081430] Lustre: Skipped 4 previous similar messages [ 2471.702243] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2471.707940] Lustre: Skipped 8 previous similar messages [ 2487.980511] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2487.992491] Lustre: Skipped 15 previous similar messages [ 2520.490737] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2520.493788] Lustre: Skipped 25 previous similar messages [ 2563.163967] Lustre: 87849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2565.795682] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:34:17 (1787852057) [ 2567.456057] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2569.317906] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:34:21 (1787852061) [ 2574.827211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.839958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2574.841485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2574.847124] Lustre: Skipped 1 previous similar message [ 2574.869933] LustreError: Skipped 1 previous similar message [ 2579.861600] Lustre: server umount lustre-OST0000 complete [ 2585.056499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.058462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2585.073163] Lustre: Skipped 1 previous similar message [ 2585.085303] Lustre: Skipped 2 previous similar messages [ 2590.177633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2594.276606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2594.292667] Lustre: Skipped 2 previous similar messages [ 2598.367244] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2598.730390] Lustre: server umount lustre-MDT0000 complete [ 2599.397225] LustreError: 79741:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.832886] LustreError: 79722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852095 with bad export cookie 7076638198724186142 [ 2602.854906] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2602.855859] LustreError: 79722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2603.473134] Lustre: server umount lustre-MDT0001 complete [ 2616.037534] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2619.419630] Key type lgssc unregistered [ 2619.790948] LNet: 89103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2619.812169] LNetError: 89103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2619.844487] LNet: Removed LNI 192.168.201.158@tcp [ 2620.909739] Key type .llcrypt unregistered [ 2620.911819] Key type ._llcrypt unregistered [ 2627.034942] Key type ._llcrypt registered [ 2627.037425] Key type .llcrypt registered [ 2627.211205] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2632.334907] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules [ 2633.806390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2633.899350] alg: No test for adler32 (adler32-zlib) [ 2635.204246] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2635.655556] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2637.495653] Key type lgssc registered [ 2639.843314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2642.010462] LustreError: 90357:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2660.973828] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2664.651591] LustreError: 90357:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2671.333508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2671.360745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.914566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.758471] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2682.348651] LustreError: 91386:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.367098] LustreError: 91386:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2687.463835] LustreError: 91385:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2688.590222] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:36:20 (1787852180) [ 2692.586541] LustreError: 91386:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.519080] LustreError: 91417:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2694.531373] LustreError: 91417:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2694.542083] LustreError: 91417:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2694.886882] Lustre: server umount lustre-MDT0000 complete [ 2713.686338] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2717.240554] Key type lgssc unregistered [ 2717.647610] LNet: 92914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2717.659563] LNetError: 92914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2718.693805] LNet: Removed LNI 192.168.201.158@tcp [ 2719.685204] Key type .llcrypt unregistered [ 2719.692896] Key type ._llcrypt unregistered [ 2745.679418] Key type ._llcrypt registered [ 2745.683909] Key type .llcrypt registered [ 2745.786302] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2747.450535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2747.490595] alg: No test for adler32 (adler32-zlib) [ 2748.706483] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2751.118229] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2752.999595] Key type lgssc registered [ 2755.215625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2766.499651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2766.513468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.006924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2773.166385] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2773.473721] LustreError: 94133:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.492656] LustreError: 94133:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2777.574884] LustreError: 94134:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.507451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.281846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2786.850477] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2794.458341] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2800.272481] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2808.490135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2809.487506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.127673] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2825.944536] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2881.487664] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:39:31 (1787852371) [ 2884.037361] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2887.131251] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:39:38 (1787852378) [ 2893.817240] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.822533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2893.848886] Lustre: Skipped 2 previous similar messages [ 2893.864943] Lustre: Skipped 2 previous similar messages [ 2897.444599] Lustre: server umount lustre-MDT0000 complete [ 2898.915236] LustreError: 94865:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.937260] LustreError: 94865:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2901.450726] LustreError: 94114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852394 with bad export cookie 13112826670960720634 [ 2901.469138] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2901.470663] LustreError: 94114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2901.930846] Lustre: server umount lustre-MDT0001 complete [ 2916.807369] Lustre: server umount lustre-OST0000 complete [ 2928.406676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.065337] LustreError: 98408:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2929.292212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.016087] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2934.760143] LustreError: 98409:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.182302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.005174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2947.948555] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2954.947277] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2960.132621] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2974.714339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2974.860915] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2974.867742] LustreError: 98398:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2974.913074] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2974.918925] LustreError: 100493:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2974.922527] LustreError: 100493:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2974.926320] LustreError: 100493:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2974.987790] Lustre: server umount lustre-OST0000 complete [ 2974.992397] LustreError: 100493:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2985.225438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2985.367729] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2985.371309] LustreError: 98398:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2985.393900] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2985.400604] LustreError: 100961:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2985.410934] LustreError: 100961:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2985.416795] LustreError: 100961:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2985.479342] Lustre: server umount lustre-OST0000 complete [ 2985.481392] LustreError: 100961:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2997.497680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2997.646349] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2997.650678] Lustre: Found index 0 for lustre-OST0000, updating log [ 2997.655949] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2997.718472] Lustre: lustre-OST0000: new disk, initializing [ 2997.722448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2997.728397] Lustre: 101497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2998.000625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3004.398291] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3008.441401] Lustre: Failing over lustre-OST0000 [ 3008.644968] Lustre: server umount lustre-OST0000 complete [ 3014.115073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3014.129626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.144237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3017.696705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3017.697693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3022.817394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3022.823732] Lustre: Skipped 2 previous similar messages [ 3022.984894] Lustre: server umount lustre-MDT0000 complete [ 3026.418115] LustreError: 98390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852519 with bad export cookie 13112826670960723378 [ 3026.445272] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3026.784996] Lustre: server umount lustre-MDT0001 complete [ 3035.105122] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:42:06 (1787852526) [ 3057.880447] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3074.746623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.365518] LustreError: 103863:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.389505] LustreError: 103863:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3075.522686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.582974] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3091.594084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3097.758548] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3101.158311] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3110.585360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3111.774782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3111.793421] Lustre: Skipped 1 previous similar message [ 3111.826430] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3116.902274] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3116.931977] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3116.963410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3121.017126] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:05 [ 3121.034392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3121.049746] Lustre: lustre-OST0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 3121.050827] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3121.077621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3122.989892] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3134.504593] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3134.589542] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3134.660866] Lustre: lustre-OST0001: new disk, initializing [ 3134.665581] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3134.673366] Lustre: 106184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3141.284567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3141.305712] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3141.517431] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3142.129314] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3150.254967] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3156.771468] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3162.085931] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.092490] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3162.104406] Lustre: Skipped 2 previous similar messages [ 3162.108941] Lustre: Skipped 1 previous similar message [ 3166.980798] Lustre: server umount lustre-MDT0001 complete [ 3167.212951] LustreError: 103864:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3167.256922] LustreError: 103864:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 3180.831977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.017898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3181.027361] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3181.177084] Lustre: 103852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3182.663154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3182.679327] Lustre: Skipped 1 previous similar message [ 3183.038443] LustreError: 107701:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3183.052989] LustreError: 107701:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3183.078351] LustreError: 103855:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3183.279445] LustreError: 107717:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.158@tcp: cfg command failed: rc = -17 [ 3183.294306] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3183.294306] [ 3187.725937] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3190.263963] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3201.590533] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:44:53 (1787852693) [ 3226.516571] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3249.349257] Lustre: 109817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3271.277380] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3274.849836] Lustre: 110951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3365.864207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3365.871440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3365.878763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3365.890744] Lustre: Skipped 2 previous similar messages [ 3366.214651] Lustre: server umount lustre-MDT0000 complete [ 3366.880804] LustreError: 111119:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.892081] LustreError: 111119:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 3369.886891] LustreError: 103845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852862 with bad export cookie 13112826670960724372 [ 3369.890413] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3369.897931] LustreError: 103845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3370.175665] Lustre: server umount lustre-MDT0001 complete [ 3384.290599] Lustre: server umount lustre-OST0000 complete [ 3387.231554] Lustre: 93446:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852864/real 1787852864] req@ffff9d398292b840 x1874698954980352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787852880 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3387.270617] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3387.288878] Lustre: Skipped 3 previous similar messages [ 3389.330148] Lustre: server umount lustre-OST0001 complete [ 3395.498960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3406.578323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3416.848261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3430.652913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3453.349919] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3465.337828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3465.369212] Lustre: lustre-MDT0000: reset Object Index mappings [ 3466.103759] LustreError: 114908:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3466.203146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.298535] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3480.142707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.161508] Lustre: lustre-MDT0001: reset Object Index mappings [ 3485.286280] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3488.182853] Lustre: 116055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3494.861866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3501.778685] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3505.868793] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3510.629522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3512.460556] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3512.470469] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3512.474691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 3519.167711] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3528.019436] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3531.825287] Lustre: 117933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3581.093650] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:51:12 (1787853072) [ 3608.548400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3608.565492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3608.578039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3620.327159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3620.690414] Lustre: server umount lustre-MDT0000 complete [ 3625.103410] LustreError: 114888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853118 with bad export cookie 13112826670960771475 [ 3625.104620] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3625.124238] LustreError: 114888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3625.442106] LustreError: 117965:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3625.464886] LustreError: 117965:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 3631.681928] Lustre: server umount lustre-MDT0001 complete [ 3642.416547] Lustre: server umount lustre-OST0000 complete [ 3653.439611] Lustre: server umount lustre-OST0001 complete [ 3663.624772] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 3676.855961] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3723.381711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3723.656748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3723.726753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3723.813665] Lustre: lustre-MDT0000: new disk, initializing [ 3723.892886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3723.904721] Lustre: Skipped 3 previous similar messages [ 3723.935826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3729.114754] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3737.394677] Lustre: server umount lustre-MDT0000 complete [ 3748.392551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3748.753119] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3748.763106] Lustre: 122206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3748.775818] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3754.977697] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3761.591209] Lustre: server umount lustre-MDT0000 complete [ 3775.191891] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:54:25 (1787853265) [ 3777.462779] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3779.433257] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:54:31 (1787853271) [ 3798.385645] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3810.047232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.267599] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3823.731123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3823.919045] Lustre: 124258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3824.001377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3824.008079] Lustre: Skipped 1 previous similar message [ 3824.139478] Lustre: lustre-MDT0001: new disk, initializing [ 3824.326227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3824.347735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3829.677264] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3834.869769] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3843.402306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3843.737527] Lustre: lustre-OST0000: new disk, initializing [ 3843.740744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3843.747832] Lustre: 125835:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3845.819813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3845.828918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3846.026621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3851.130494] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3863.244301] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3863.475338] Lustre: lustre-OST0001: new disk, initializing [ 3863.480960] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3863.488520] Lustre: 126706:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3865.515660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3865.541443] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3865.769076] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3871.813202] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3883.432909] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3887.438928] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4262.887395] Lustre: 133134:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4262.902469] Lustre: 133134:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4262.914855] Lustre: 133134:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4262.927677] Lustre: 133134:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4262.935966] Lustre: 133134:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4262.943891] Lustre: 133134:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4546.873922] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:07:18 (1787854038) [ 4567.020665] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4567.032849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4567.048899] Lustre: Skipped 8 previous similar messages [ 4567.065154] Lustre: Skipped 19 previous similar messages [ 4569.062410] Lustre: server umount lustre-MDT0000 complete [ 4572.137984] LustreError: 127523:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4572.158819] LustreError: 127523:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 4572.597782] LustreError: 126705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854065 with bad export cookie 13112826670960838332 [ 4572.628778] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4577.258960] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4577.263936] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4577.279484] Lustre: Skipped 1 previous similar message [ 4577.289043] Lustre: Skipped 1 previous similar message [ 4579.279750] Lustre: server umount lustre-MDT0001 complete [ 4589.339500] Lustre: server umount lustre-OST0000 complete [ 4599.169848] Lustre: server umount lustre-OST0001 complete [ 4607.119173] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 4618.071894] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4653.801861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4654.015265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4654.033115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4654.096201] Lustre: lustre-MDT0000: new disk, initializing [ 4654.160279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.165972] Lustre: Skipped 5 previous similar messages [ 4654.181455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4658.166183] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4667.898834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.049898] Lustre: 136185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4668.117245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4668.125654] Lustre: Skipped 1 previous similar message [ 4668.219856] Lustre: lustre-MDT0001: new disk, initializing [ 4668.385532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4668.395326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4672.365565] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4682.476398] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4688.909940] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4701.406853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4702.051297] Lustre: lustre-OST0000: new disk, initializing [ 4702.062944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4702.079569] Lustre: 138152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4704.015245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4704.035673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4704.247297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4710.375590] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4719.240715] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4728.215322] Lustre: lustre-MDD0000: changelog on [ 4730.754452] Lustre: lustre-MDD0001: changelog on [ 6426.539040] Lustre: lustre-MDD0001: changelog off [ 6429.014410] Lustre: 136191:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6429.791216] Lustre: lustre-MDD0000: changelog off [ 6439.910447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6439.911448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6439.915657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6439.928683] Lustre: Skipped 1 previous similar message [ 6439.962360] LustreError: Skipped 1 previous similar message [ 6445.029038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6445.039169] Lustre: Skipped 2 previous similar messages [ 6450.159525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6450.169803] Lustre: Skipped 1 previous similar message [ 6451.169298] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6451.402538] Lustre: server umount lustre-OST0000 complete [ 6460.390160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6460.390377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6460.418203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6460.432263] Lustre: Skipped 1 previous similar message [ 6462.252831] Lustre: server umount lustre-MDT0000 complete [ 6465.508977] LustreError: 136192:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6465.533653] LustreError: 136192:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6466.495477] LustreError: 136176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855959 with bad export cookie 13112826670961661728 [ 6466.499180] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6466.502837] LustreError: 136176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6467.018875] Lustre: server umount lustre-MDT0001 complete [ 6478.760576] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 6481.824896] Key type lgssc unregistered [ 6482.110972] LNet: 147256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6482.123259] LNetError: 147256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6482.138182] LNet: Removed LNI 192.168.201.158@tcp [ 6483.110361] Key type .llcrypt unregistered [ 6483.113147] Key type ._llcrypt unregistered [ 6505.093952] Key type ._llcrypt registered [ 6505.096360] Key type .llcrypt registered [ 6505.208744] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 6522.118794] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 6523.592504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6523.621810] alg: No test for adler32 (adler32-zlib) [ 6524.670166] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 6524.841330] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 6526.479441] Key type lgssc registered [ 6527.649044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6562.782265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6562.816332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6564.093556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6564.114587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6564.258463] Lustre: lustre-MDT0000: new disk, initializing [ 6564.326436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6564.363832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6569.520789] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6579.045717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6579.158931] Lustre: 150457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6579.203309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6579.206667] Lustre: Skipped 1 previous similar message [ 6579.266878] Lustre: lustre-MDT0001: new disk, initializing [ 6579.368382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6579.417430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6579.437307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6583.740672] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6592.930325] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6598.181558] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6604.796113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6605.078404] Lustre: lustre-OST0000: new disk, initializing [ 6605.081745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6605.087890] Lustre: 152427:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6605.203874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6611.326936] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6612.546138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6612.559190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6612.696078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6622.923482] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6631.279910] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_module obdecho/obdecho [ 6633.347675] LustreError: 153431:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6633.356306] LustreError: 153431:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6634.959646] LustreError: 153527:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6648.801087] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6648.822662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6648.831223] Lustre: Skipped 2 previous similar messages [ 6648.850691] Lustre: Skipped 2 previous similar messages [ 6655.685470] Lustre: server umount lustre-MDT0000 complete [ 6659.048782] LustreError: 150466:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6659.078839] LustreError: 150466:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 6659.626683] LustreError: 150451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856152 with bad export cookie 8631423604123024479 [ 6659.632623] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6659.643652] LustreError: 150451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6660.306199] Lustre: server umount lustre-MDT0001 complete [ 6674.705820] Lustre: server umount lustre-OST0000 complete [ 6697.676971] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 6710.725784] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 6745.768095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.004276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6746.057385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6746.166893] Lustre: lustre-MDT0000: new disk, initializing [ 6746.249392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6746.280509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6751.576663] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6762.367943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6762.497925] Lustre: 157242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6762.570848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6762.580191] Lustre: Skipped 1 previous similar message [ 6762.592593] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6762.733343] Lustre: lustre-MDT0001: new disk, initializing [ 6762.886467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6762.925096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6762.940337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6768.342989] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6777.659197] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6783.965845] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6790.536298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6790.865430] Lustre: lustre-OST0000: new disk, initializing [ 6790.874610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6790.889787] Lustre: 159211:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6791.046363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6798.054075] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6798.457357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6798.470390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6798.615617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6808.061586] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6813.014657] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6813.230286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6818.005121] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6818.199996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6824.420305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6824.421381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6824.455374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6824.467371] Lustre: Skipped 1 previous similar message [ 6826.320911] Lustre: server umount lustre-OST0000 complete [ 6834.656072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6834.656809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6834.657916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6834.680125] LustreError: Skipped 1 previous similar message [ 6834.682763] Lustre: Skipped 2 previous similar messages [ 6836.644613] Lustre: server umount lustre-MDT0000 complete [ 6839.779179] LustreError: 157256:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6839.802138] LustreError: 157256:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6840.856721] LustreError: 157235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856333 with bad export cookie 8631423604123025438 [ 6840.864766] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6840.866728] LustreError: 157235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6841.503872] Lustre: server umount lustre-MDT0001 complete [ 6845.007795] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:45:36 (1787856336) [ 6846.802966] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6848.857561] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:45:40 (1787856340) [ 6856.704713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6857.128663] LustreError: 161625:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6857.205663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.934682] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6862.318279] LustreError: 161626:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6867.431199] LustreError: 161625:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6870.016778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6875.479655] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6884.028658] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6890.600506] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6898.410796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6898.614369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6898.621846] Lustre: Skipped 1 previous similar message [ 6905.417539] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6913.681267] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6924.529329] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.58@tcp (stopping) [ 6924.536240] Lustre: Skipped 1 previous similar message [ 6925.285300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6925.296571] LustreError: 162612:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6925.297138] Lustre: Skipped 1 previous similar message [ 6925.320386] Lustre: server umount lustre-MDT0000 complete [ 6929.224173] LustreError: 161606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856422 with bad export cookie 8631423604123026320 [ 6929.236784] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6929.237589] LustreError: 161606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6930.411084] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6934.434707] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6934.449605] Lustre: Skipped 2 previous similar messages [ 6935.824423] Lustre: server umount lustre-MDT0001 complete [ 6942.636274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6957.406709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6957.671690] LustreError: 165465:0:(llog.c:1655:llog_backup()) MGC192.168.201.158@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6957.675833] Lustre: 165465:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.158@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6959.074093] LustreError: 165470:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6959.083745] LustreError: 165470:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 6994.912241] LustreError: 165470:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.934082] LustreError: 165470:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 26 previous similar messages [ 7018.464443] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7018.482210] Lustre: 165465:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.158@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7059.457789] LustreError: 165471:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.464651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7059.481029] LustreError: 165471:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 45 previous similar messages [ 7064.569136] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7064.676649] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7073.937474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7074.802361] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x77c8fa0965acdcc6 [ 7074.821575] Lustre: MGC192.168.201.158@tcp: Connection restored to 0@lo (at 0@lo) [ 7079.986760] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7080.433359] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7087.687248] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7089.393735] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7095.114555] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7096.567772] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7121.380119] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7121.381718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7121.423653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7121.423662] Lustre: Skipped 1 previous similar message [ 7125.752991] Lustre: server umount lustre-MDT0000 complete [ 7130.193318] LustreError: 161606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856623 with bad export cookie 8631423604123032774 [ 7130.199611] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7130.201703] LustreError: 161606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7130.782672] Lustre: server umount lustre-MDT0001 complete [ 7145.692303] Lustre: server umount lustre-OST0000 complete [ 7167.126893] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 7176.871872] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 7215.667201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7215.968026] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7216.002370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7216.104825] Lustre: lustre-MDT0000: new disk, initializing [ 7216.225836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7216.232060] Lustre: Skipped 1 previous similar message [ 7216.244627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7221.431190] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7233.969615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7234.130241] Lustre: 170575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7234.436095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7239.742874] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7250.301990] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7256.139231] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7263.689567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7264.035336] Lustre: lustre-OST0000: new disk, initializing [ 7264.037832] Lustre: Skipped 1 previous similar message [ 7264.042249] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7264.046458] Lustre: Skipped 2 previous similar messages [ 7264.051050] Lustre: 172543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7270.959318] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7271.500674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7271.507482] Lustre: Skipped 1 previous similar message [ 7271.519051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7271.621410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7281.052470] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7286.032314] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7286.309719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7291.376591] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7291.710807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7297.507041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7297.511799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7297.516888] LustreError: Skipped 1 previous similar message [ 7297.523953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7297.533796] Lustre: Skipped 3 previous similar messages [ 7297.568989] Lustre: Skipped 3 previous similar messages [ 7299.899597] Lustre: server umount lustre-OST0000 complete [ 7306.208237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7310.567245] Lustre: server umount lustre-MDT0000 complete [ 7312.864502] LustreError: 170580:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7312.895284] LustreError: 170580:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 16 previous similar messages [ 7314.592996] LustreError: 170565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856807 with bad export cookie 8631423604123049609 [ 7314.595759] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7314.604061] LustreError: 170565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7314.836583] Lustre: server umount lustre-MDT0001 complete [ 7322.954972] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:53:34 (1787856814) [ 7329.725735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.698763] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7344.404857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7349.858161] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7357.405971] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7362.832193] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7370.318555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7377.809169] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7386.141913] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7402.634592] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:54:54 (1787856894) [ 7409.632860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7409.649833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7409.660796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7409.660803] Lustre: Skipped 3 previous similar messages [ 7409.694734] Lustre: Skipped 3 previous similar messages [ 7414.259815] Lustre: server umount lustre-OST0000 complete [ 7427.957322] Lustre: server umount lustre-MDT0000 complete [ 7432.750919] LustreError: 174937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856925 with bad export cookie 8631423604123050491 [ 7432.754070] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7432.760941] LustreError: 174937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7433.136455] Lustre: server umount lustre-MDT0001 complete [ 7445.796835] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 7448.752068] Key type lgssc unregistered [ 7449.064922] LNet: 178691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7449.071993] LNetError: 178691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7449.094271] LNet: Removed LNI 192.168.201.158@tcp [ 7449.866335] Key type .llcrypt unregistered [ 7449.869681] Key type ._llcrypt unregistered [ 7467.727374] Key type ._llcrypt registered [ 7467.730264] Key type .llcrypt registered [ 7467.843883] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 7469.193767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7469.266718] alg: No test for adler32 (adler32-zlib) [ 7470.533313] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 7470.821894] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 7472.495181] Key type lgssc registered [ 7473.895714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7481.332496] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7481.343928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7496.863544] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7501.983753] LustreError: 179933:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [ 7507.104547] LustreError: 179933:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7507.111463] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7507.117282] LustreError: MGC192.168.201.158@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7507.127261] LustreError: 179933:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7507.140752] LustreError: 179933:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7507.153978] LustreError: 179933:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7507.198971] Lustre: server umount lustre-OST0000 complete [ 7507.202256] LustreError: 179933:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7516.326657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7517.823760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7522.073590] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7527.402980] LustreError: 180434:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7527.415896] LustreError: 180434:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7530.708837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7531.306534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7536.128344] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7542.898393] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7548.322585] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7554.812775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7555.069438] LustreError: 182251:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7555.090295] Lustre: 182251:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.158@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7555.374611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7562.981399] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7571.393705] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7577.569429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7577.584462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7577.628890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7578.086046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.111373] Lustre: Skipped 1 previous similar message [ 7583.202442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7583.213614] Lustre: Skipped 5 previous similar messages [ 7588.322103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7589.343548] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7589.731354] Lustre: server umount lustre-MDT0000 complete [ 7593.441163] LustreError: 180433:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7593.474810] LustreError: 180433:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7593.908792] LustreError: 180415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857086 with bad export cookie 1393189388673612737 [ 7593.910796] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7593.917945] LustreError: 180415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7594.177172] Lustre: server umount lustre-MDT0001 complete [ 7608.483340] Lustre: server umount lustre-OST0000 complete [ 7620.137922] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 7630.223682] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 7672.238729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.536547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7672.562812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7672.646378] Lustre: lustre-MDT0000: new disk, initializing [ 7672.767934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7672.794513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7677.452257] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7687.967869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.071574] Lustre: 185781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7688.130436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7688.134526] Lustre: Skipped 1 previous similar message [ 7688.259574] Lustre: lustre-MDT0001: new disk, initializing [ 7688.412413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7688.466506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7688.477309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7692.817197] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7702.176131] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7708.397129] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7715.627957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7716.058191] Lustre: lustre-OST0000: new disk, initializing [ 7716.061577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7716.075654] Lustre: 187749:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7716.191539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7723.162638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7723.181784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7723.304286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7723.395536] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7733.623699] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7739.337319] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7739.547458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7745.664058] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7745.931679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7749.088465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7749.096452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7749.115575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7749.137188] Lustre: Skipped 1 previous similar message [ 7749.149836] Lustre: Skipped 3 previous similar messages [ 7754.233154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7754.243754] Lustre: Skipped 1 previous similar message [ 7762.405704] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7762.769361] Lustre: server umount lustre-OST0000 complete [ 7769.571353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7769.575637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7769.608898] Lustre: Skipped 2 previous similar messages [ 7769.609820] Lustre: Skipped 1 previous similar message [ 7773.089640] Lustre: server umount lustre-MDT0000 complete [ 7774.688734] LustreError: 185792:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7776.823827] LustreError: 185772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857269 with bad export cookie 1393189388673613465 [ 7776.836965] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7776.841644] LustreError: 185772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7777.352377] Lustre: server umount lustre-MDT0001 complete [ 7786.276458] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:01:18 (1787857278) [ 7805.012950] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 7808.286562] Key type lgssc unregistered [ 7808.587492] LNet: 190710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7808.606364] LNetError: 190710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7809.644745] LNet: Removed LNI 192.168.201.158@tcp [ 7810.589844] Key type .llcrypt unregistered [ 7810.591941] Key type ._llcrypt unregistered [ 7829.431157] Key type ._llcrypt registered [ 7829.441207] Key type .llcrypt registered [ 7829.565249] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 7830.653524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7830.689848] alg: No test for adler32 (adler32-zlib) [ 7831.873803] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 7832.124802] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 7833.823616] Key type lgssc registered [ 7835.417681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7844.037345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7844.066598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7845.587549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7850.978339] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7855.077836] LustreError: 191931:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7855.107715] LustreError: 191931:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7860.193024] LustreError: 191932:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7861.158517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7861.823670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7867.013913] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7875.784698] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7882.754672] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7892.960699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7893.248760] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7893.251181] LustreError: 193795:0:(llog.c:1696:llog_backup()) MGC192.168.201.158@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7893.259954] Lustre: 193795:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.158@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7893.417266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7900.364084] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7909.271745] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7912.050580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7912.070188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7912.082339] Lustre: Skipped 2 previous similar messages [ 7917.540660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7917.548328] Lustre: Skipped 2 previous similar messages [ 7918.448080] Lustre: server umount lustre-MDT0000 complete [ 7922.012567] LustreError: 193823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857415 with bad export cookie 115400929877584291 [ 7922.013930] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7922.023780] LustreError: 193823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7922.318781] Lustre: server umount lustre-MDT0001 complete [ 7936.894371] Lustre: server umount lustre-OST0000 complete [ 7947.373585] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 7958.628617] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 7997.862122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.108181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7998.135662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7998.196318] Lustre: lustre-MDT0000: new disk, initializing [ 7998.265196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7998.279727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8003.307840] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8013.838522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8013.960775] Lustre: 197321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8013.999094] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8014.000844] Lustre: Skipped 1 previous similar message [ 8014.006563] LustreError: 198103:0:(llog.c:1696:llog_backup()) MGC192.168.201.158@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8014.011649] LustreError: 198103:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8014.015253] Lustre: 198103:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.158@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8014.028884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8014.032338] Lustre: Skipped 1 previous similar message [ 8014.113100] Lustre: lustre-MDT0001: new disk, initializing [ 8014.176166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8014.233887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8014.243125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8018.738169] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8027.844539] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8033.514920] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8039.921682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8040.129936] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8040.132531] Lustre: Skipped 1 previous similar message [ 8040.135430] LustreError: 199265:0:(llog.c:1696:llog_backup()) MGC192.168.201.158@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8040.139536] LustreError: 199265:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8040.146119] Lustre: 199265:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.158@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8040.150336] Lustre: 199265:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8040.164458] Lustre: lustre-OST0000: new disk, initializing [ 8040.167812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8040.174683] Lustre: 199288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8040.250294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8046.504462] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8046.678704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8046.686435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8046.961460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8056.787885] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8061.965060] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8062.245853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8066.886483] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8067.160654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8072.699497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8072.709319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8072.728523] Lustre: Skipped 2 previous similar messages [ 8072.732660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8075.156990] Lustre: server umount lustre-OST0000 complete [ 8080.865502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8080.874534] LustreError: Skipped 1 previous similar message [ 8080.878879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8080.908376] Lustre: Skipped 1 previous similar message [ 8080.932713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8080.935233] Lustre: Skipped 1 previous similar message [ 8082.935710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8086.654784] Lustre: server umount lustre-MDT0000 complete [ 8088.033039] LustreError: 198782:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8088.071469] LustreError: 198782:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8091.078114] LustreError: 197313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857584 with bad export cookie 115400929877585005 [ 8091.080390] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8091.090465] LustreError: 197313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8091.663803] Lustre: server umount lustre-MDT0001 complete [ 8100.296512] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 15:06:32 (1787857592) [ 8116.953536] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8127.141361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8127.469122] LustreError: 202415:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8127.538758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8132.375471] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8132.585393] LustreError: 202416:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8141.392305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8146.421383] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8149.305732] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8157.116995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8163.211503] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8172.182014] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8172.306260] Lustre: lustre-OST0001: new disk, initializing [ 8172.308926] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8172.313457] Lustre: 204726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8172.399422] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8172.406610] Lustre: Skipped 2 previous similar messages [ 8179.401720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8179.414733] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8179.553596] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8179.721280] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8190.774452] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8198.609709] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8200.795738] Lustre: DEBUG MARKER: ADD OST3 [ 8207.008080] loop: module loaded [ 8207.153312] loop0: detected capacity change from 0 to 204800000 [ 8207.326071] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8207.341755] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8207.361396] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8213.903980] loop0: detected capacity change from 0 to 204800000 [ 8214.758862] Lustre: DEBUG MARKER: STOP OST3 [ 8220.875671] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8221.214930] Lustre: lustre-OST0003: new disk, initializing [ 8221.225870] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8221.237884] Lustre: 206745:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8222.871160] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8223.045065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8223.053321] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8223.106601] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8224.936048] LustreError: 206861:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8225.270850] Lustre: Failing over lustre-OST0003 [ 8225.407960] Lustre: server umount lustre-OST0003 complete [ 8227.599799] LustreError: 203904:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8227.634193] LustreError: 203904:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8228.329329] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8228.350179] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8229.152577] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8239.231678] Lustre: DEBUG MARKER: START OST3 again [ 8243.714357] LustreError: 205056:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8243.736875] LustreError: 205056:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 8246.757033] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8247.132310] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8247.145126] Lustre: Skipped 1 previous similar message [ 8247.166637] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8248.017995] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8248.822651] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8248.823358] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8254.821244] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8278.151883] Lustre: Failing over lustre-OST0003 [ 8278.311942] Lustre: server umount lustre-OST0003 complete [ 8279.528829] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8279.538258] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8279.556967] Lustre: Skipped 1 previous similar message [ 8279.564249] LustreError: 205056:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.580050] LustreError: 205056:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8300.516650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8300.531115] LustreError: Skipped 1 previous similar message [ 8300.534906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.561310] Lustre: Skipped 1 previous similar message [ 8300.576785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8300.582570] Lustre: Skipped 2 previous similar messages [ 8306.313211] Lustre: server umount lustre-MDT0000 complete [ 8311.475043] LustreError: 202397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857804 with bad export cookie 115400929877585887 [ 8311.480816] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8311.498709] LustreError: 202397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8313.831449] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8313.840205] Lustre: Skipped 5 previous similar messages [ 8318.203034] Lustre: server umount lustre-MDT0001 complete [ 8323.543598] Lustre: server umount lustre-OST0000 complete [ 8329.785497] Lustre: server umount lustre-OST0001 complete [ 8337.905722] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 8348.746978] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8385.353785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.595427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8385.627032] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8385.734923] Lustre: lustre-MDT0000: new disk, initializing [ 8385.869681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8385.882496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8390.134111] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8400.926521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8401.091953] Lustre: 212203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8401.357211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8405.939422] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8416.737610] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8423.637120] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8433.625541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8434.206994] Lustre: lustre-OST0000: new disk, initializing [ 8434.219640] Lustre: Skipped 1 previous similar message [ 8434.233722] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8434.249987] Lustre: Skipped 2 previous similar messages [ 8434.281673] Lustre: 214172:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8435.417807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8435.433377] Lustre: Skipped 1 previous similar message [ 8435.442604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8435.581070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8442.951880] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8454.414795] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8459.350528] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8459.621240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8465.778872] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8466.052950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8471.520148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8471.525968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8471.534496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8471.537418] Lustre: Skipped 6 previous similar messages [ 8471.545132] Lustre: Skipped 2 previous similar messages [ 8474.523461] Lustre: server umount lustre-OST0000 complete [ 8484.644171] Lustre: server umount lustre-MDT0000 complete [ 8486.880760] LustreError: 212210:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8486.893604] LustreError: 212210:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 19 previous similar messages [ 8488.702767] LustreError: 212194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857981 with bad export cookie 115400929877588183 [ 8488.715411] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8488.728848] LustreError: 212194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8489.092780] Lustre: server umount lustre-MDT0001 complete [ 8492.593126] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 15:13:04 (1787857984) [ 8512.339602] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 8523.436333] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8568.844319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8569.163638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8569.205303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8569.329857] Lustre: lustre-MDT0000: new disk, initializing [ 8569.518908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8574.392591] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8585.098385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8585.281605] Lustre: 218787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8585.535789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8590.235370] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8600.166320] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8606.210576] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8613.639229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8613.964649] Lustre: 220756:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8615.471496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8621.505728] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8631.850657] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8637.276176] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8637.524673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8642.370560] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8642.618422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8646.113300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8646.146658] LustreError: Skipped 1 previous similar message [ 8646.148816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8646.166238] Lustre: Skipped 2 previous similar messages [ 8646.170507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8646.173116] Lustre: Skipped 3 previous similar messages [ 8651.246272] Lustre: server umount lustre-OST0000 complete [ 8669.151131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8669.547559] Lustre: server umount lustre-MDT0000 complete [ 8671.717421] LustreError: 218793:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8671.738085] LustreError: 218793:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8675.490844] LustreError: 218779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858168 with bad export cookie 115400929877589065 [ 8675.501539] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8675.993662] Lustre: server umount lustre-MDT0001 complete [ 8688.028236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.534097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8688.542643] Lustre: Skipped 5 previous similar messages [ 8694.665565] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8703.749068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8709.876827] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8718.233294] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8724.760977] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8732.762647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8739.319567] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8746.863354] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8759.777914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8759.790167] LustreError: Skipped 2 previous similar messages [ 8759.796702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8759.819227] Lustre: Skipped 8 previous similar messages [ 8763.621618] Lustre: server umount lustre-OST0000 complete [ 8782.304198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8782.645194] Lustre: server umount lustre-MDT0000 complete [ 8786.885350] LustreError: 225851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858279 with bad export cookie 115400929877589947 [ 8786.885773] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8786.899300] LustreError: 225851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8787.271563] Lustre: server umount lustre-MDT0001 complete [ 8886.141838] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 8888.632170] Key type lgssc unregistered [ 8888.874370] LNet: 226641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8888.880890] LNetError: 226641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8888.895690] LNet: Removed LNI 192.168.201.158@tcp [ 8889.630201] Key type .llcrypt unregistered [ 8889.637443] Key type ._llcrypt unregistered [ 8922.780120] Key type ._llcrypt registered [ 8922.781568] Key type .llcrypt registered [ 8922.870323] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 8941.026711] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8942.509324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8942.557239] alg: No test for adler32 (adler32-zlib) [ 8943.712447] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 8944.065682] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 8945.815269] Key type lgssc registered [ 8947.065415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8986.595393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8986.656815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8988.083608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8988.181598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8988.347774] Lustre: lustre-MDT0000: new disk, initializing [ 8988.515152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8988.549763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8992.977684] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9002.766416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9002.898529] Lustre: 230005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9002.968565] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9002.980447] Lustre: Skipped 1 previous similar message [ 9003.102408] Lustre: lustre-MDT0001: new disk, initializing [ 9003.191767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9003.224045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9003.233453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9007.933379] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9016.365539] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9021.801762] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9030.041397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9030.316762] Lustre: lustre-OST0000: new disk, initializing [ 9030.319882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9030.325689] Lustre: 231973:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9030.394568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9036.216285] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9036.356730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9036.363535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9036.517229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9045.483250] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9049.853171] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9050.049734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9054.063230] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9054.306895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9057.259053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9057.263977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9057.273194] LustreError: Skipped 1 previous similar message [ 9057.280793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9057.286463] Lustre: Skipped 1 previous similar message [ 9057.311227] Lustre: Skipped 1 previous similar message [ 9062.276459] Lustre: server umount lustre-OST0000 complete [ 9067.491528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9067.496894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9067.502943] Lustre: Skipped 1 previous similar message [ 9067.513065] Lustre: Skipped 1 previous similar message [ 9072.577249] Lustre: server umount lustre-MDT0000 complete [ 9076.327656] LustreError: 229998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858569 with bad export cookie 17669341383758316237 [ 9076.330137] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9076.345316] LustreError: 229998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9076.675726] Lustre: server umount lustre-MDT0001 complete [ 9095.644320] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9105.836703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9106.143099] LustreError: 234905:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9106.166425] LustreError: 234905:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9106.208316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9110.860616] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9111.525499] LustreError: 234906:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9116.642376] LustreError: 234905:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9118.516806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9118.827731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9123.355821] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9126.817422] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9133.839168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9134.062540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9140.930135] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9151.914841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9152.015252] Lustre: lustre-OST0001: new disk, initializing [ 9152.017253] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9152.020247] Lustre: 237210:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9152.066708] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9157.713042] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9161.391448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9161.410784] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9161.583835] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9169.364238] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9175.458772] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9186.633152] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 15:24:38 (1787858678) [ 9190.370901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9190.384644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9190.402266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9192.423641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9192.427175] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9192.433051] Lustre: Skipped 1 previous similar message [ 9192.461153] Lustre: Skipped 2 previous similar messages [ 9196.596197] Lustre: server umount lustre-MDT0000 complete [ 9197.550330] LustreError: 234901:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9197.576536] LustreError: 234901:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9199.818598] LustreError: 234887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858692 with bad export cookie 17669341383758317119 [ 9199.825310] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9199.833969] LustreError: 234887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9200.125467] Lustre: server umount lustre-MDT0001 complete [ 9213.739717] Lustre: server umount lustre-OST0000 complete [ 9226.928799] Lustre: server umount lustre-OST0001 complete [ 9232.821685] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 9242.478725] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9276.483616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9276.810819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9276.883511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9277.010388] Lustre: lustre-MDT0000: new disk, initializing [ 9277.151361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9277.171716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9281.377543] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9291.608711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9291.699602] Lustre: 241154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9291.806755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9291.815147] Lustre: Skipped 1 previous similar message [ 9291.986986] Lustre: lustre-MDT0001: new disk, initializing [ 9292.246892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9292.257511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9297.028306] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9306.944991] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9312.785846] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9321.109154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9321.548177] Lustre: lustre-OST0000: new disk, initializing [ 9321.551303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9321.558032] Lustre: 243125:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9323.167538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9323.181961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9323.311623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9328.404360] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9338.827680] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9344.005869] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9344.343067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9349.903163] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9350.208260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9354.216984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9354.227887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9354.230799] LustreError: Skipped 1 previous similar message [ 9354.234243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9354.250124] Lustre: Skipped 1 previous similar message [ 9354.273211] Lustre: Skipped 3 previous similar messages [ 9364.448951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9364.466048] Lustre: Skipped 3 previous similar messages [ 9367.007431] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9367.315972] Lustre: server umount lustre-OST0000 complete [ 9374.181160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9374.198502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9377.721471] Lustre: server umount lustre-MDT0000 complete [ 9379.822665] LustreError: 241168:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9379.844402] LustreError: 241168:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9382.317523] LustreError: 241148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858875 with bad export cookie 17669341383758319247 [ 9382.319051] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9382.323387] LustreError: 241148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9382.812663] Lustre: server umount lustre-MDT0001 complete [ 9391.272233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9391.606880] LustreError: 245341:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9391.688417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9391.694479] Lustre: Skipped 2 previous similar messages [ 9396.060937] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9405.458609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9410.193653] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9419.433154] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9425.870696] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9433.117874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9439.451104] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9448.369223] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9594.338795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9594.362108] Lustre: Skipped 1 previous similar message [ 9594.390118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9594.410106] Lustre: Skipped 3 previous similar messages [ 9600.292500] Lustre: server umount lustre-OST0000 complete [ 9610.664218] Lustre: server umount lustre-MDT0000 complete [ 9614.771292] LustreError: 245322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859107 with bad export cookie 17669341383758320136 [ 9614.786076] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9615.187274] Lustre: server umount lustre-MDT0001 complete [ 9717.688558] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 9721.285589] Key type lgssc unregistered [ 9721.679797] LNet: 248955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9721.689537] LNetError: 248955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9722.727662] LNet: Removed LNI 192.168.201.158@tcp [ 9723.836239] Key type .llcrypt unregistered [ 9723.839479] Key type ._llcrypt unregistered [ 9749.421539] Key type ._llcrypt registered [ 9749.423092] Key type .llcrypt registered [ 9749.523354] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 9763.082299] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9763.965313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9764.042064] alg: No test for adler32 (adler32-zlib) [ 9765.185666] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 9765.430495] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 9767.103153] Key type lgssc registered [ 9768.365346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9813.677452] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9824.627018] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9824.660526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9826.076094] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9826.095756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9826.155832] Lustre: lustre-MDT0000: new disk, initializing [ 9826.258158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9826.275377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9831.486096] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9843.685785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.784835] Lustre: 253028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9843.836126] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9843.842894] Lustre: Skipped 1 previous similar message [ 9843.899185] Lustre: lustre-MDT0001: new disk, initializing [ 9843.975045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9844.009732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9844.021877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9848.603946] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9854.537442] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9861.551971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9861.725319] Lustre: lustre-OST0000: new disk, initializing [ 9861.727885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9861.735222] Lustre: 254660:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9861.819079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9865.527215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9865.535804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9865.623991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9868.728092] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9882.545205] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9882.815982] Lustre: lustre-OST0001: new disk, initializing [ 9882.828114] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9882.840300] Lustre: 255532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9882.914347] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9889.976464] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9890.353645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9890.364420] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9890.459584] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9901.069422] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9910.368909] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9920.993927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9921.002362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9921.017179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9921.514232] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9921.535761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9921.538360] Lustre: Skipped 1 previous similar message [ 9922.681740] Lustre: server umount lustre-MDT0000 complete [ 9925.915441] LustreError: 253018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859418 with bad export cookie 7301277406509970899 [ 9925.915799] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9925.924280] LustreError: 253018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9926.499498] Lustre: server umount lustre-MDT0001 complete [ 9940.422273] Lustre: server umount lustre-OST0000 complete [ 9942.515896] Lustre: 250532:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859419/real 1787859419] req@ffff9d3abb995680 x1874706309394944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787859435 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9942.555962] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9942.577510] Lustre: Skipped 2 previous similar messages [ 9944.433889] Lustre: server umount lustre-OST0001 complete [ 9958.863372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9968.603571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9976.447401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9985.348610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10005.944989] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [10015.523154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10015.544297] Lustre: lustre-MDT0000: reset Object Index mappings [10015.920221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10020.357758] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10021.348489] LustreError: 260540:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10021.365446] LustreError: 260540:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10025.454584] LustreError: 260541:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10028.207153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.240406] Lustre: lustre-MDT0001: reset Object Index mappings [10028.670173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10033.294895] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10036.236245] Lustre: 261683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10043.139720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10049.918296] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10053.801652] LustreError: 262036:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10059.237973] LustreError: 262035:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10059.464328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10059.696561] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10059.700171] Lustre: Skipped 1 previous similar message [10060.728963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10060.742620] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10066.871424] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10075.133829] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10078.703543] Lustre: 263554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10084.839871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10084.856518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10084.883402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10084.888036] Lustre: Skipped 1 previous similar message [10086.370762] LustreError: 261272:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10086.399880] LustreError: 261272:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10086.436545] Lustre: server umount lustre-MDT0000 complete [10089.992263] LustreError: 260522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859582 with bad export cookie 7301277406509977696 [10090.000851] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10090.007363] LustreError: 260522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10090.326566] Lustre: server umount lustre-MDT0001 complete [10104.536235] Lustre: server umount lustre-OST0000 complete [10107.680192] Lustre: 250532:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859584/real 1787859584] req@ffff9d3ab0050000 x1874706309443456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787859600 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10107.704817] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.713888] Lustre: Skipped 3 previous similar messages [10108.571568] Lustre: server umount lustre-OST0001 complete [10127.925941] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:40:19 (1787859619) [10144.098224] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [10157.092409] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [10192.942784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10193.182726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10193.219343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10193.305363] Lustre: lustre-MDT0000: new disk, initializing [10193.389900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10193.402584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10198.563042] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10210.300484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10210.455568] Lustre: 267808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10210.535369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10210.539500] Lustre: Skipped 1 previous similar message [10210.661457] Lustre: lustre-MDT0001: new disk, initializing [10210.861534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10210.869348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10217.248878] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10229.112842] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10235.356359] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10242.149731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10242.595662] Lustre: lustre-OST0000: new disk, initializing [10242.601465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10242.607230] Lustre: 269779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10244.300830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10244.323904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10244.536777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10250.262600] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10259.906370] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10264.563413] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10264.801582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10269.702300] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10269.991177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10275.309505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10275.313959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10275.322897] LustreError: Skipped 1 previous similar message [10275.328556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10275.369521] Lustre: Skipped 1 previous similar message [10278.077750] Lustre: server umount lustre-OST0000 complete [10282.464473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10282.475793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10282.482793] Lustre: Skipped 1 previous similar message [10290.657142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10290.659774] Lustre: Skipped 3 previous similar messages [10296.287462] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10296.512962] Lustre: server umount lustre-MDT0000 complete [10300.898871] LustreError: 267821:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10300.915619] LustreError: 267821:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10301.113963] LustreError: 267803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859794 with bad export cookie 7301277406509979789 [10301.121779] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10301.130991] LustreError: 267803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10301.577745] Lustre: server umount lustre-MDT0001 complete [10320.459496] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [10330.544592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10330.820576] LustreError: 272715:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10330.882526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10330.891630] Lustre: Skipped 2 previous similar messages [10335.410885] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10343.914807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10348.960807] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10351.783575] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10358.537604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10364.570192] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10373.071410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10373.366682] Lustre: lustre-OST0001: new disk, initializing [10373.371976] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10373.393441] Lustre: 275023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10374.552121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10374.560265] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10374.686531] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10380.077682] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10390.925997] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10395.302758] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10402.813299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10402.884400] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10410.982162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10410.989980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10411.008196] Lustre: Skipped 2 previous similar messages [10411.026287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10411.039408] Lustre: Skipped 2 previous similar messages [10416.419422] Lustre: server umount lustre-MDT0000 complete [10416.628950] LustreError: 272711:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10416.662304] LustreError: 272711:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10420.577732] LustreError: 272696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859913 with bad export cookie 7301277406509980671 [10420.579320] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10420.591242] LustreError: 272696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10420.977779] Lustre: server umount lustre-MDT0001 complete [10435.879864] Lustre: server umount lustre-OST0000 complete [10437.603666] Lustre: 250533:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787859914/real 1787859914] req@ffff9d3aa4b192c0 x1874706309555712/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787859930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10440.391262] Lustre: server umount lustre-OST0001 complete [10459.657498] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [10470.742302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10471.236986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10471.246024] Lustre: Skipped 3 previous similar messages [10476.136166] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10481.645184] LustreError: 278081:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10481.675216] LustreError: 278081:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10485.055810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10490.662331] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10494.163329] Lustre: 279222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10501.606529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10509.195771] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10517.557445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10520.192441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10525.703346] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10526.725657] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10533.939058] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10537.505701] Lustre: 281101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10543.303325] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10543.390337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10555.688513] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:47:26 (1787860046) [10571.754069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10571.766968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10571.776271] Lustre: Skipped 5 previous similar messages [10571.789679] Lustre: Skipped 7 previous similar messages [10577.854473] Lustre: server umount lustre-MDT0000 complete [10581.615794] LustreError: 278061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860074 with bad export cookie 7301277406509987202 [10581.620211] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10581.623877] LustreError: 278061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10588.032966] Lustre: server umount lustre-MDT0001 complete [10598.780848] Lustre: server umount lustre-OST0000 complete [10603.274241] Lustre: server umount lustre-OST0001 complete [10610.215577] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [10620.929061] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [10671.139120] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [10681.088371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.334795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10681.363898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10681.464648] Lustre: lustre-MDT0000: new disk, initializing [10681.584434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10686.442427] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10697.607137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10697.810555] Lustre: 285339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10698.118329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10702.964356] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10715.710664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10716.041537] Lustre: 286977:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10717.241356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10722.595380] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10733.488860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10733.701796] Lustre: 287847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10733.808474] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10733.811204] Lustre: Skipped 6 previous similar messages [10735.798882] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10735.820019] Lustre: Skipped 1 previous similar message [10735.938486] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10738.868722] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10748.535457] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10752.534721] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10752.539725] Lustre: Skipped 1 previous similar message [10771.886134] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:51:03 (1787860263) [10775.008925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10775.029775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10775.050094] Lustre: Skipped 4 previous similar messages [10775.062459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10775.081705] Lustre: Skipped 8 previous similar messages [10776.656323] Lustre: server umount lustre-MDT0000 complete [10777.059164] LustreError: 285352:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10777.084821] LustreError: 285352:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [10780.953824] LustreError: 285333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860273 with bad export cookie 7301277406509994839 [10780.966268] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10780.968813] LustreError: 285333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10781.391055] Lustre: server umount lustre-MDT0001 complete [10796.609961] Lustre: server umount lustre-OST0000 complete [10798.371576] Lustre: 250534:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787860275/real 1787860275] req@ffff9d3abcc61680 x1874706309707008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787860291 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10800.776205] Lustre: server umount lustre-OST0001 complete [10810.672493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10816.785909] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10825.892484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10831.601466] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10838.793595] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10844.974539] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10851.563586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10858.619616] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10869.607020] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10890.402544] loop1: detected capacity change from 0 to 204800000 [10890.787664] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10890.803584] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10890.817468] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10897.499317] loop1: detected capacity change from 0 to 204800000 [10904.835138] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10904.998191] LustreError: 290086:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs2571: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10905.023816] LustreError: 290086:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs2571 [10905.032681] LustreError: 290086:0:(mgs_handler.c:517:mgs_target_reg()) fs2571-MDT0001: Error checking target index: rc = -13 [10905.054667] LustreError: fs2571-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10905.072784] LustreError: 293816:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10905.084838] LustreError: 293816:0:(tgt_mount.c:2129:server_put_super()) no obd fs2571-MDT0001 [10905.098232] LustreError: 293816:0:(tgt_mount.c:116:server_deregister_mount()) fs2571-MDT0001 not registered [10905.151444] Lustre: server umount fs2571-MDT0001 complete [10905.154864] LustreError: 293816:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10917.429177] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10917.686507] Lustre: Setting parameter fs2571-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs2571-MDT0001 [10917.784726] Lustre: srv-fs2571-MDT0001: No data found on store. Initialize space. [10917.805626] Lustre: Skipped 4 previous similar messages [10918.019437] Lustre: fs2571-MDT0001: new disk, initializing [10918.024873] Lustre: Skipped 3 previous similar messages [10918.288870] Lustre: srv-fs2571-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10918.300611] Lustre: 294283:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2571-MDT0001-mdtlov: can't create new llog: rc = -115 [10918.324223] LustreError: 294283:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2571-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10923.922621] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10931.759402] Lustre: server umount fs2571-MDT0001 complete [10934.772758] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10945.742399] loop1: detected capacity change from 0 to 204800000 [10951.934279] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10952.276485] Lustre: 295852:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2571-MDT0001-mdtlov: can't create new llog: rc = -115 [10952.297555] LustreError: 295852:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2571-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10958.047904] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10963.497375] Lustre: server umount fs2571-MDT0001 complete [10975.716281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10975.717501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10975.734615] Lustre: Skipped 6 previous similar messages [10975.739894] Lustre: Skipped 1 previous similar message [10981.755697] Lustre: server umount lustre-MDT0000 complete [10991.771604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10991.999770] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10997.493846] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10997.767736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11005.958251] loop1: detected capacity change from 0 to 204800000 [11012.461459] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11013.106884] Lustre: 298299:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2571-MDT0001-mdtlov: can't create new llog: rc = -115 [11013.121954] LustreError: 298299:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2571-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11018.398351] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11023.826418] Lustre: server umount fs2571-MDT0001 complete [11033.598145] LustreError: 291923:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11033.634085] LustreError: 291923:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 112 previous similar messages [11051.491914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11051.516890] LustreError: Skipped 1 previous similar message [11065.311735] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11065.697639] Lustre: server umount lustre-OST0000 complete [11076.613771] Lustre: server umount lustre-MDT0000 complete [11081.058300] LustreError: 290077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860574 with bad export cookie 7301277406509997520 [11081.073443] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11081.081809] LustreError: 290077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11081.880403] Lustre: server umount lustre-MDT0001 complete [11098.314551] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [11101.052702] Key type lgssc unregistered [11101.381653] LNet: 300651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11101.386748] LNetError: 300651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11101.417669] LNet: Removed LNI 192.168.201.158@tcp [11102.208106] Key type .llcrypt unregistered [11102.214271] Key type ._llcrypt unregistered [11126.649917] Key type ._llcrypt registered [11126.654203] Key type .llcrypt registered [11126.800332] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [11127.885187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11127.917513] alg: No test for adler32 (adler32-zlib) [11128.981779] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [11129.342034] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [11131.031328] Key type lgssc registered [11132.778079] Lustre: Echo OBD driver; http://www.lustre.org/ [11142.180332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11142.204845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11143.883082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11148.284721] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11153.377395] LustreError: 301924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11153.394154] LustreError: 301924:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11158.025950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11158.458434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11164.142629] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11171.621912] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11177.551324] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11184.337795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11184.794800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11188.915160] LustreError: 303756:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11192.953440] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11194.354631] LustreError: 303754:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11199.471967] LustreError: 304187:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11202.096129] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11203.567694] LustreError: 304187:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11203.604027] LustreError: 304187:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11213.814807] LustreError: 303756:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11213.846449] LustreError: 303756:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11223.918596] loop1: detected capacity change from 0 to 204800000 [11224.172558] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11224.183857] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11224.204499] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11229.627473] loop1: detected capacity change from 0 to 204800000 [11229.905954] LustreError: 303755:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11229.929040] LustreError: 303755:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [11235.899375] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11236.101981] Lustre: fs12474-OST0000: new disk, initializing [11236.105532] Lustre: srv-fs12474-OST0000: No data found on store. Initialize space. [11236.112723] Lustre: 305662:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11236.207212] Lustre: fs12474-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11242.697888] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11251.900452] Lustre: server umount fs12474-OST0000 complete [11263.472463] loop1: detected capacity change from 0 to 204800000 [11265.004466] LustreError: 303754:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11265.017119] LustreError: 303754:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 18 previous similar messages [11268.071896] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11270.194393] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11270.632156] Lustre: fs12474-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11278.244380] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11283.436788] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11288.798698] loop2: detected capacity change from 0 to 204800000 [11289.091698] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11289.100876] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11289.112587] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11294.761489] loop2: detected capacity change from 0 to 204800000 [11301.591638] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11301.645097] LustreError: fs12474-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11301.650749] LustreError: fs12474-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11301.658444] LustreError: 308701:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11301.667611] LustreError: 308701:0:(tgt_mount.c:2129:server_put_super()) no obd fs12474-OST0001 [11301.675210] LustreError: 308701:0:(tgt_mount.c:116:server_deregister_mount()) fs12474-OST0001 not registered [11301.737233] Lustre: server umount fs12474-OST0001 complete [11301.742845] LustreError: 308701:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11313.960157] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11314.329611] Lustre: fs12474-OST0001: new disk, initializing [11314.333638] Lustre: srv-fs12474-OST0001: No data found on store. Initialize space. [11314.344479] Lustre: 309252:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11314.482123] Lustre: fs12474-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11321.390425] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11329.521375] LustreError: 303754:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11329.555206] LustreError: 303754:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 39 previous similar messages [11331.539850] Lustre: server umount fs12474-OST0000 complete [11343.830226] Lustre: server umount fs12474-OST0001 complete [11359.636867] loop1: detected capacity change from 0 to 204800000 [11367.066108] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11367.427906] Lustre: fs12474-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11374.105880] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11381.071215] loop2: detected capacity change from 0 to 204800000 [11387.434530] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11394.007551] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11405.049849] Lustre: server umount fs12474-OST0000 complete [11416.542605] Lustre: server umount fs12474-OST0001 complete [11421.678702] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11433.953914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11433.959337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11433.979551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11434.977641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11434.999948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11437.026870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11437.046983] Lustre: Skipped 1 previous similar message [11442.153299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11442.167066] Lustre: Skipped 1 previous similar message [11446.240843] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11446.397172] Lustre: server umount lustre-OST0000 complete [11450.353818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11450.359756] LustreError: Skipped 2 previous similar messages [11450.365502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11450.374261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11452.384317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11456.841281] Lustre: server umount lustre-MDT0000 complete [11460.765019] LustreError: 301904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860953 with bad export cookie 5175482548759954290 [11460.777800] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11461.467086] Lustre: server umount lustre-MDT0001 complete [11472.646373] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [11475.520165] Key type lgssc unregistered [11475.794782] LNet: 315194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11475.805138] LNetError: 315194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11475.821659] LNet: Removed LNI 192.168.201.158@tcp [11476.601230] Key type .llcrypt unregistered [11476.602936] Key type ._llcrypt unregistered [11511.276823] Key type ._llcrypt registered [11511.291653] Key type .llcrypt registered [11511.474726] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [11530.419825] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [11531.558684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11531.799102] alg: No test for adler32 (adler32-zlib) [11532.925249] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [11533.205928] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [11534.935203] Key type lgssc registered [11536.170221] Lustre: Echo OBD driver; http://www.lustre.org/ [11570.186229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11570.218957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11571.448069] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11571.478769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11571.553872] Lustre: lustre-MDT0000: new disk, initializing [11571.647560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11571.681721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11576.098032] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11586.476577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11586.624881] Lustre: 319056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11586.709475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11586.718423] Lustre: Skipped 1 previous similar message [11586.890467] Lustre: lustre-MDT0001: new disk, initializing [11587.053671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11587.093213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11587.104662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11591.472222] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11600.284742] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11606.058691] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11612.998160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11613.228873] Lustre: lustre-OST0000: new disk, initializing [11613.235688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11613.245257] Lustre: 321026:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11613.421886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11621.228290] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11621.477051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11621.484375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11621.707659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11632.390769] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11634.738083] Lustre: Mounted lustre-client - version 2.17.57_98_gaf12cc4 [11636.110848] Lustre: Unmounted lustre-client [11647.470117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11647.474400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11647.488337] LustreError: Skipped 1 previous similar message [11647.501841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11647.517572] Lustre: Skipped 1 previous similar message [11650.024798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11650.033027] Lustre: Skipped 1 previous similar message [11655.150811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11660.259244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11660.283011] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11660.294863] Lustre: Skipped 2 previous similar messages [11660.517315] Lustre: server umount lustre-OST0000 complete [11665.377531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11665.392861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11665.397878] Lustre: Skipped 2 previous similar messages [11671.082841] Lustre: server umount lustre-MDT0000 complete [11676.595948] LustreError: 319049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861169 with bad export cookie 1186118505990893855 [11676.607829] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11676.619841] LustreError: 319049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11677.103940] Lustre: server umount lustre-MDT0001 complete [11689.310494] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [11690.911777] LNet: 1 local NIs in recovery (showing 1): 192.168.201.158@tcp [11692.567206] Key type lgssc unregistered [11692.902407] LNet: 323191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11692.935505] LNetError: 323191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11765.672250] LNet: Removed LNI 192.168.201.158@tcp [11766.455795] Key type .llcrypt unregistered [11766.458457] Key type ._llcrypt unregistered [11786.602584] Key type ._llcrypt registered [11786.605590] Key type .llcrypt registered [11786.796627] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [11809.069903] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [11810.649229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11810.713485] alg: No test for adler32 (adler32-zlib) [11811.839918] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [11812.175041] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [11813.839255] Key type lgssc registered [11815.241610] Lustre: Echo OBD driver; http://www.lustre.org/ [11850.272043] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:09:02 (1787861342) [11856.239557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11856.260469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11857.635170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11857.679562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11857.819655] Lustre: lustre-MDT0000: new disk, initializing [11857.956617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11858.012708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11862.242145] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11872.787943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11872.960968] Lustre: 326415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11873.013979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11873.018215] Lustre: Skipped 1 previous similar message [11873.109333] Lustre: lustre-MDT0001: new disk, initializing [11873.183751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11873.214990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11873.232803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11878.506507] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11887.946409] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11893.956219] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11901.613382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11901.949338] Lustre: lustre-OST0000: new disk, initializing [11901.957345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11901.964068] Lustre: 328384:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11902.052067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11909.391378] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11912.272797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11912.289985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11912.422880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11919.118537] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11938.294748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11938.295448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11938.299905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11938.302101] LustreError: Skipped 1 previous similar message [11938.338495] Lustre: Skipped 1 previous similar message [11941.653767] Lustre: server umount lustre-OST0000 complete [11948.512968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11948.529307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11948.538570] Lustre: Skipped 1 previous similar message [11953.635811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11953.645364] Lustre: Skipped 1 previous similar message [11958.752862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11958.762522] Lustre: Skipped 1 previous similar message [11960.299822] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11960.619447] Lustre: server umount lustre-MDT0000 complete [11964.846655] LustreError: 326407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861457 with bad export cookie 9535576262797338291 [11964.859046] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11964.862274] LustreError: 326407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11965.232158] Lustre: server umount lustre-MDT0001 complete [11976.526812] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [11979.321392] Key type lgssc unregistered [11979.626228] LNet: 330297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11979.641118] LNetError: 330297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11979.686958] LNet: Removed LNI 192.168.201.158@tcp [11980.497615] Key type .llcrypt unregistered [11980.503643] Key type ._llcrypt unregistered [12004.594729] Key type ._llcrypt registered [12004.604285] Key type .llcrypt registered [12004.830811] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [12007.199258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12007.520171] alg: No test for adler32 (adler32-zlib) [12008.795842] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [12009.267173] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [12011.055925] Key type lgssc registered [12012.919709] Lustre: Echo OBD driver; http://www.lustre.org/ [12022.071182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12022.086135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12023.538116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12028.450444] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12034.032362] LustreError: 331571:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12034.055980] LustreError: 331571:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12037.328562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12037.826905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12043.721484] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12052.377447] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12058.834437] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12067.516618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12067.860313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12074.149876] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12082.298517] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12093.287988] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12094.814591] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12099.042500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12099.056107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12099.081805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12100.328373] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.58@tcp (stopping) [12102.628161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12102.631519] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12102.643085] Lustre: Skipped 1 previous similar message [12102.672994] Lustre: Skipped 1 previous similar message [12105.144792] Lustre: server umount lustre-MDT0000 complete [12105.423907] LustreError: 334488:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12107.748555] LustreError: 331568:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12107.781075] LustreError: 331568:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12108.950816] LustreError: 331553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861601 with bad export cookie 4992025922582039692 [12108.961346] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12108.963727] LustreError: 331553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12110.544963] LustreError: 333786:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12110.583976] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.58@tcp (stopping) [12110.590044] Lustre: Skipped 1 previous similar message [12112.865952] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12115.782041] Lustre: server umount lustre-MDT0001 complete [12120.013810] Lustre: server umount lustre-OST0000 complete [12130.059670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12134.567870] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12138.118510] Lustre: 335753:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.158@tcp [12140.382591] Lustre: server umount MGS complete [12149.564033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12150.170932] LustreError: 336326:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12150.203292] LustreError: 336326:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12150.291355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12156.023190] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12160.482101] LustreError: 336326:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12160.498714] LustreError: 336326:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12165.635621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12165.843876] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.58@tcp (not set up) [12165.849715] Lustre: Skipped 2 previous similar messages [12166.105729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12170.771274] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12180.365681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12180.593855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12187.166155] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12196.475425] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12198.569472] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12204.762711] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12206.495329] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12211.680423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12211.685408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12211.692180] LustreError: Skipped 1 previous similar message [12211.700369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12213.479686] Lustre: server umount lustre-OST0000 complete [12221.923118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12221.944605] Lustre: Skipped 2 previous similar messages [12224.834404] Lustre: server umount lustre-MDT0000 complete [12227.040511] LustreError: 336326:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12227.060285] LustreError: 336326:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12228.628163] LustreError: 338723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861721 with bad export cookie 4992025922582040532 [12228.630568] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12228.645743] LustreError: 338723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12229.037963] Lustre: server umount lustre-MDT0001 complete [12240.946932] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12244.593776] Key type lgssc unregistered [12244.947760] LNet: 339765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12244.958628] LNetError: 339765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12244.978903] LNet: Removed LNI 192.168.201.158@tcp [12245.928387] Key type .llcrypt unregistered [12245.941119] Key type ._llcrypt unregistered [12266.086651] Key type ._llcrypt registered [12266.103085] Key type .llcrypt registered [12266.248421] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [12284.520836] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [12285.646584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12285.679714] alg: No test for adler32 (adler32-zlib) [12287.122560] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [12287.453669] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [12289.200133] Key type lgssc registered [12291.255639] Lustre: Echo OBD driver; http://www.lustre.org/ [12334.246769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12334.291486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12335.566573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12335.610814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12335.717044] Lustre: lustre-MDT0000: new disk, initializing [12335.840167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12335.866365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12340.745425] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12354.224723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12354.366375] Lustre: 342926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12354.437102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12354.442736] Lustre: Skipped 1 previous similar message [12354.538308] Lustre: lustre-MDT0001: new disk, initializing [12354.680138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12354.734321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12354.744943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12359.335162] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12369.669571] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12374.755352] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12383.085324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12383.305499] Lustre: lustre-OST0000: new disk, initializing [12383.310070] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12383.317961] Lustre: 344894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12383.402711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12390.036509] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12391.074238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12391.091737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12391.288376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12400.816935] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12406.281808] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12406.433455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12411.134755] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12411.403789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12416.999774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12417.016625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12417.041060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12417.043438] Lustre: Skipped 1 previous similar message [12421.179675] Lustre: server umount lustre-OST0000 complete [12426.212886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12426.227851] LustreError: Skipped 1 previous similar message [12426.247885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12426.289978] Lustre: Skipped 1 previous similar message [12426.306089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12432.752401] Lustre: server umount lustre-MDT0000 complete [12437.475825] LustreError: 342932:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12437.508902] LustreError: 342932:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12437.964263] LustreError: 342917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861930 with bad export cookie 11501736119412984800 [12437.979297] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12437.988181] LustreError: 342917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12438.896735] Lustre: server umount lustre-MDT0001 complete [12448.674498] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:19:00 (1787861940) [12463.930165] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [12473.511970] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [12508.938305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12509.245570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12509.275348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12509.365312] Lustre: lustre-MDT0000: new disk, initializing [12509.483635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12509.501267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12514.467721] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12525.380158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12525.517901] Lustre: 349661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12525.557078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12525.561170] Lustre: Skipped 1 previous similar message [12525.676871] Lustre: lustre-MDT0001: new disk, initializing [12525.825190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12525.862536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12525.880521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12531.165500] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12541.230953] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12546.940323] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12553.663129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12554.099684] Lustre: lustre-OST0000: new disk, initializing [12554.113020] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12554.122704] Lustre: 351628:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12554.246369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12562.606960] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12564.031113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12564.046102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12564.243974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12573.175833] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12620.793967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12620.799484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12620.806710] Lustre: Skipped 1 previous similar message [12620.816682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12620.818737] Lustre: Skipped 3 previous similar messages [12624.045056] Lustre: server umount lustre-OST0000 complete [12628.449558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12628.455872] LustreError: Skipped 1 previous similar message [12628.459361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12628.469291] Lustre: Skipped 1 previous similar message [12628.477172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12634.921172] Lustre: server umount lustre-MDT0000 complete [12636.130439] LustreError: 349674:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12636.172314] LustreError: 349674:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12639.592493] LustreError: 349652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862132 with bad export cookie 11501736119412985682 [12639.594100] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12639.604234] LustreError: 349652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12639.975618] Lustre: server umount lustre-MDT0001 complete [12653.653576] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12657.291220] Key type lgssc unregistered [12657.623638] LNet: 353807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12657.640906] LNetError: 353807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12657.685088] LNet: Removed LNI 192.168.201.158@tcp [12658.812103] Key type .llcrypt unregistered [12658.816149] Key type ._llcrypt unregistered [12747.864159] Key type ._llcrypt registered [12747.867407] Key type .llcrypt registered [12748.025639] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12748.470970] Key type .llcrypt unregistered [12748.477240] Key type ._llcrypt unregistered [12829.300819] Key type ._llcrypt registered [12829.307253] Key type .llcrypt registered [12829.438267] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12829.756731] Key type .llcrypt unregistered [12829.759081] Key type ._llcrypt unregistered [12855.994217] Key type ._llcrypt registered [12855.996204] Key type .llcrypt registered [12856.145969] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12856.410449] Key type .llcrypt unregistered [12856.412680] Key type ._llcrypt unregistered [12884.675945] Key type ._llcrypt registered [12884.679333] Key type .llcrypt registered [12884.834296] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12885.343819] Key type .llcrypt unregistered [12885.349559] Key type ._llcrypt unregistered [12980.589190] Key type ._llcrypt registered [12980.593469] Key type .llcrypt registered [12980.800358] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12981.550347] Key type .llcrypt unregistered [12981.554046] Key type ._llcrypt unregistered [13015.198373] Key type ._llcrypt registered [13015.199798] Key type .llcrypt registered [13015.328697] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13015.659585] Key type .llcrypt unregistered [13015.663090] Key type ._llcrypt unregistered [13046.572070] Key type ._llcrypt registered [13046.575931] Key type .llcrypt registered [13046.684925] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13046.968768] Key type .llcrypt unregistered [13046.973454] Key type ._llcrypt unregistered [13081.303619] Key type ._llcrypt registered [13081.305894] Key type .llcrypt registered [13081.477839] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13081.884324] Key type .llcrypt unregistered [13081.887429] Key type ._llcrypt unregistered [13110.806363] Key type ._llcrypt registered [13110.811436] Key type .llcrypt registered [13110.921935] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13111.161356] Key type .llcrypt unregistered [13111.165600] Key type ._llcrypt unregistered [13139.638946] Key type ._llcrypt registered [13139.641630] Key type .llcrypt registered [13139.728842] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13140.008285] Key type .llcrypt unregistered [13140.010526] Key type ._llcrypt unregistered [13204.723942] Key type ._llcrypt registered [13204.727403] Key type .llcrypt registered [13204.856246] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13205.163294] Key type .llcrypt unregistered [13205.165625] Key type ._llcrypt unregistered [13237.770802] Key type ._llcrypt registered [13237.773235] Key type .llcrypt registered [13237.865496] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13238.108588] Key type .llcrypt unregistered [13238.111714] Key type ._llcrypt unregistered [13267.525842] Key type ._llcrypt registered [13267.529620] Key type .llcrypt registered [13267.685238] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13268.082502] Key type .llcrypt unregistered [13268.084215] Key type ._llcrypt unregistered [13330.703553] Key type ._llcrypt registered [13330.705434] Key type .llcrypt registered [13330.805683] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [13350.996306] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [13351.721326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13351.748517] alg: No test for adler32 (adler32-zlib) [13352.907564] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [13353.105480] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [13354.759971] Key type lgssc registered [13356.001315] Lustre: Echo OBD driver; http://www.lustre.org/ [13388.079185] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13388.103616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13389.392922] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13389.419841] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13389.473687] Lustre: tst250fs-MDT0000: new disk, initializing [13389.587074] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13389.603986] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13394.307837] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13404.046114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13404.208717] Lustre: 374169:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13404.256544] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13404.260705] Lustre: Skipped 1 previous similar message [13404.321246] Lustre: tst250fs-MDT0001: new disk, initializing [13404.419912] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13404.455514] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13404.466588] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13409.129596] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13414.992650] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13423.251410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13423.504807] Lustre: tst250fs-OST0000: new disk, initializing [13423.512548] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13423.522916] Lustre: 375800:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13423.624827] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13430.686410] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13433.998187] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13434.009960] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13434.139440] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13441.303202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13441.549990] Lustre: tst250fs-OST0001: new disk, initializing [13441.554905] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13441.559528] Lustre: 376673:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13441.661365] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13441.905109] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13441.917828] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13442.003101] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13449.219853] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13454.421446] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13456.545473] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13456.553349] Lustre: Skipped 1 previous similar message [13477.868892] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13477.872464] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13477.898995] Lustre: Skipped 3 previous similar messages [13477.902603] Lustre: Skipped 2 previous similar messages [13482.994888] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13483.001359] Lustre: Skipped 2 previous similar messages [13483.711470] Lustre: server umount tst250fs-MDT0000 complete [13487.811421] LustreError: 374163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862980 with bad export cookie 17639215796640984039 [13487.840097] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13488.356875] Lustre: server umount tst250fs-MDT0001 complete [13502.423411] Lustre: server umount tst250fs-OST0000 complete [13516.498712] Lustre: server umount tst250fs-OST0001 complete [13537.456561] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [13547.133990] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [13584.518412] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:37:55 (1787863075) [13586.832740] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13606.246393] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [13619.452066] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [13657.528995] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13350 sec ========= 16:39:09 (1787863149) [13660.062255] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:39:11 (1787863151) === [13663.846357] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:39:15 (1787863155) === [13711.898320] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13715.387980] Key type lgssc unregistered [13715.770938] LNet: 385811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13715.782184] LNetError: 385811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13715.803921] LNet: Removed LNI 192.168.201.158@tcp [13717.158631] Key type .llcrypt unregistered [13717.161272] Key type ._llcrypt unregistered