[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508984989 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003044] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008446] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011144] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.015006] *** VALIDATE selinux *** [ 0.023649] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028302] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032040] *** VALIDATE tmpfs *** [ 0.033496] *** VALIDATE proc *** [ 0.034243] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.036282] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.042714] debug: unmapping init [mem 0xffffffffbbe59000-0xffffffffbbe60fff] [ 0.044851] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045722] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053331] rcu: Hierarchical SRCU implementation. [ 0.055456] smp: Bringing up secondary CPUs ... [ 0.056630] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.061140] smp: Brought up 1 node, 4 CPUs [ 0.063038] smpboot: Max logical packages: 1 [ 0.064021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.112147] node 0 deferred pages initialised in 46ms [ 0.117301] devtmpfs: initialized [ 0.120844] x86/mm: Memory block size: 128MB [ 0.128408] gcov: version magic: 0x41383552 [ 0.132301] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.133076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137759] pinctrl core: initialized pinctrl subsystem [ 0.141345] [ 0.141947] ************************************************************* [ 0.146017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149012] ** ** [ 0.152012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155016] ** ** [ 0.157011] ** This means that this kernel is built to expose internal ** [ 0.160015] ** IOMMU data structures, which may compromise security on ** [ 0.162011] ** your system. ** [ 0.165025] ** ** [ 0.168016] ** If you see this message and you are not debugging the ** [ 0.171013] ** kernel, report this immediately to your vendor! ** [ 0.174013] ** ** [ 0.177020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180011] ************************************************************* [ 0.183709] NET: Registered protocol family 16 [ 0.185488] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.191061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194141] cpuidle: using governor menu [ 0.196512] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.199454] PCI: Using configuration type 1 for base access [ 0.202146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209154] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.210046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.213095] cryptd: max_cpu_qlen set to 1000 [ 0.216269] ACPI: Added _OSI(Module Device) [ 0.218015] ACPI: Added _OSI(Processor Device) [ 0.219013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.221014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.225900] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.231398] ACPI: Interpreter enabled [ 0.233054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.234012] ACPI: Using IOAPIC for interrupt routing [ 0.236075] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239431] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.249964] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.253045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.258077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.263092] acpiphp: Slot [2] registered [ 0.264095] acpiphp: Slot [5] registered [ 0.265091] acpiphp: Slot [6] registered [ 0.266152] acpiphp: Slot [7] registered [ 0.268085] acpiphp: Slot [8] registered [ 0.269140] acpiphp: Slot [9] registered [ 0.270160] acpiphp: Slot [10] registered [ 0.273233] acpiphp: Slot [3] registered [ 0.275193] acpiphp: Slot [4] registered [ 0.277336] acpiphp: Slot [11] registered [ 0.279327] acpiphp: Slot [12] registered [ 0.281114] acpiphp: Slot [13] registered [ 0.282098] acpiphp: Slot [14] registered [ 0.284118] acpiphp: Slot [15] registered [ 0.285081] acpiphp: Slot [16] registered [ 0.286075] acpiphp: Slot [17] registered [ 0.287097] acpiphp: Slot [18] registered [ 0.288039] acpiphp: Slot [19] registered [ 0.288968] acpiphp: Slot [20] registered [ 0.290075] acpiphp: Slot [21] registered [ 0.291072] acpiphp: Slot [22] registered [ 0.292082] acpiphp: Slot [23] registered [ 0.294126] acpiphp: Slot [24] registered [ 0.295080] acpiphp: Slot [25] registered [ 0.296088] acpiphp: Slot [26] registered [ 0.297094] acpiphp: Slot [27] registered [ 0.299091] acpiphp: Slot [28] registered [ 0.300078] acpiphp: Slot [29] registered [ 0.302100] acpiphp: Slot [30] registered [ 0.303144] acpiphp: Slot [31] registered [ 0.305091] PCI host bridge to bus 0000:00 [ 0.306018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.308016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.310017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.313015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.314023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.316265] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.319048] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.323339] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.332016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.336494] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.339017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.341016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.343012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345509] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.348715] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.351026] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.352520] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.357012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.369015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.371011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.374185] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.386017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.402018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.411126] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.417018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.423018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.437021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.444956] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.455017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.464018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.482042] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.494038] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.501032] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.507014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.521017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.531080] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.544019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.551019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.569016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.578391] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.585017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.596020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.620019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.636506] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.638366] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.641368] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.643368] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.646208] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.650283] iommu: Default domain type: Passthrough [ 0.652595] SCSI subsystem initialized [ 0.655173] ACPI: bus type USB registered [ 0.657123] usbcore: registered new interface driver usbfs [ 0.659099] usbcore: registered new interface driver hub [ 0.662076] usbcore: registered new device driver usb [ 0.663171] pps_core: LinuxPPS API ver. 1 registered [ 0.665009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.667069] PTP clock support registered [ 0.669055] EDAC MC: Ver: 3.0.0 [ 0.671128] PCI: Using ACPI for IRQ routing [ 0.673009] NetLabel: Initializing [ 0.675012] NetLabel: domain hash size = 128 [ 0.676009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.678082] NetLabel: unlabeled traffic allowed by default [ 0.680004] vgaarb: loaded [ 0.681357] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.682000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.687000] clocksource: Switched to clocksource kvm-clock [ 0.795200] VFS: Disk quotas dquot_6.6.0 [ 0.796651] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.799023] *** VALIDATE ramfs *** [ 0.800171] *** VALIDATE hugetlbfs *** [ 0.802137] pnp: PnP ACPI init [ 0.804387] pnp: PnP ACPI: found 6 devices [ 0.825738] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.828825] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.830919] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.832898] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.835170] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.837428] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.840134] NET: Registered protocol family 2 [ 0.842459] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.847467] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.851093] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.856286] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.859375] TCP: Hash tables configured (established 65536 bind 65536) [ 0.861819] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.864303] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.866475] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.868946] NET: Registered protocol family 1 [ 0.872409] RPC: Registered named UNIX socket transport module. [ 0.874370] RPC: Registered udp transport module. [ 0.875620] RPC: Registered tcp transport module. [ 0.876951] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.878799] NET: Registered protocol family 44 [ 0.880041] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.881613] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.883304] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.885216] PCI: CLS 0 bytes, default 64 [ 0.886415] Unpacking initramfs... [ 2.264445] debug: unmapping init [mem 0xffffa00b3cc54000-0xffffa00b3ffbffff] [ 2.268554] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.270865] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.273913] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.761204] Initialise system trusted keyrings [ 2.762814] Key type blacklist registered [ 2.765484] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.774883] zbud: loaded [ 2.779873] *** VALIDATE nfs *** [ 2.781242] *** VALIDATE nfs4 *** [ 2.782982] pstore: using deflate compression [ 2.788312] Platform Keyring initialized [ 2.899838] NET: Registered protocol family 38 [ 2.901818] Key type asymmetric registered [ 2.903344] Asymmetric key parser 'x509' registered [ 2.905228] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.907990] io scheduler mq-deadline registered [ 2.909847] io scheduler kyber registered [ 2.911819] io scheduler bfq registered [ 2.914203] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.917463] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.920605] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.923801] ACPI: Power Button [PWRF] [ 2.929257] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.936203] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.949664] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.955118] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.974675] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.001335] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.029449] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.035054] Non-volatile memory driver v1.3 [ 3.036432] Linux agpgart interface v0.103 [ 3.073644] virtio_blk virtio1: [vda] 147040 512-byte logical blocks (75.3 MB/71.8 MiB) [ 3.076762] vda: detected capacity change from 0 to 75284480 [ 3.099408] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.101418] vdb: detected capacity change from 0 to 1073741824 [ 3.124208] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.126833] vdc: detected capacity change from 0 to 2621440000 [ 3.140418] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.143432] vdd: detected capacity change from 0 to 2621440000 [ 3.158459] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.161091] vde: detected capacity change from 0 to 4294967296 [ 3.186646] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.189457] vdf: detected capacity change from 0 to 4294967296 [ 3.195404] libphy: Fixed MDIO Bus: probed [ 3.200899] usbcore: registered new interface driver usbserial_generic [ 3.203244] usbserial: USB Serial support registered for generic [ 3.205430] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.209597] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.211539] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.214152] mousedev: PS/2 mouse device common for all mice [ 3.217394] rtc_cmos 00:05: RTC can wake from S4 [ 3.218813] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.221758] rtc_cmos 00:05: registered as rtc0 [ 3.223632] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.226896] intel_pstate: CPU model not supported [ 3.229499] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.232743] hid: raw HID events driver (C) Jiri Kosina [ 3.235052] usbcore: registered new interface driver usbhid [ 3.235081] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.236721] usbhid: USB HID core driver [ 3.236895] drop_monitor: Initializing network drop monitor service [ 3.243036] Initializing XFRM netlink socket [ 3.245204] NET: Registered protocol family 10 [ 3.247894] Segment Routing with IPv6 [ 3.249229] NET: Registered protocol family 17 [ 3.251197] mpls_gso: MPLS GSO support [ 3.258314] RAS: Correctable Errors collector initialized. [ 3.259726] AVX version of gcm_enc/dec engaged. [ 3.261102] AES CTR mode by8 optimization enabled [ 3.333992] sched_clock: Marking stable (3333967783, 0)->(4307714984, -973747201) [ 3.337742] registered taskstats version 1 [ 3.339935] Loading compiled-in X.509 certificates [ 3.341257] zswap: loaded using pool lzo/zbud [ 3.362191] Key type big_key registered [ 3.372771] Key type encrypted registered [ 3.374297] ima: No TPM chip found, activating TPM-bypass! [ 3.376291] ima: Allocated hash algorithm: sha1 [ 3.377838] ima: No architecture policies found [ 3.379510] evm: Initialising EVM extended attributes: [ 3.381431] evm: security.selinux [ 3.382639] evm: security.ima [ 3.383689] evm: security.capability [ 3.384903] evm: HMAC attrs: 0x1 [ 3.387218] rtc_cmos 00:05: setting system clock to 2026-08-07 07:57:38 UTC (1786089458) [ 3.394127] debug: unmapping init [mem 0xffffffffbce03000-0xffffffffbcffffff] [ 3.396731] debug: unmapping init [mem 0xffffffffbbb82000-0xffffffffbbe58fff] [ 3.407121] Write protecting the kernel read-only data: 28672k [ 3.410497] debug: unmapping init [mem 0xffffffffba203000-0xffffffffba3fffff] [ 3.413055] debug: unmapping init [mem 0xffffffffbab14000-0xffffffffbabfffff] [ 3.445559] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.454480] systemd[1]: Detected virtualization kvm. [ 3.456407] systemd[1]: Detected architecture x86-64. [ 3.458494] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.485279] systemd[1]: No hostname configured. [ 3.486327] systemd[1]: Set hostname to . [ 3.487580] random: systemd: uninitialized urandom read (16 bytes read) [ 3.489052] systemd[1]: Initializing machine ID from random generator. [ 3.532663] random: ln: uninitialized urandom read (6 bytes read) [ 3.602556] random: systemd: uninitialized urandom read (16 bytes read) [ 3.605799] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.612525] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.615383] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.097312] device-mapper: uevent: version 1.0.3 [ 4.099882] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.701623] random: fast init done [ 4.738125] virtio_net virtio0 ens2: renamed from eth0 [ 4.822824] scsi host0: ata_piix [ 4.868645] scsi host1: ata_piix [ 4.869909] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.871876] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.351189] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.889204] random: crng init done [ 9.890493] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.081504] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.072374] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.286983] SELinux: Disabled at runtime. [ 11.346343] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.353157] systemd[1]: Detected virtualization kvm. [ 11.354524] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.914861] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.920315] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.927192] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.931614] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.934853] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.942032] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.949915] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 12.124177] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.371219] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.671870] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.740225] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.795164] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.804957] EDAC sbridge: Ver: 1.1.2 [ 14.131550] Key type dns_resolver registered [ 14.412789] NFS: Registering the id_resolver key type [ 14.414921] Key type id_resolver registered [ 14.416700] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg106-server login: [ 31.025224] hrtimer: interrupt took 2490613 ns [ 75.671823] libcfs: loading out-of-tree module taints kernel. [ 75.715710] Key type ._llcrypt registered [ 75.719090] Key type .llcrypt registered [ 75.866547] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 97.106973] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 98.790807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.811735] alg: No test for adler32 (adler32-zlib) [ 100.229364] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 101.210989] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 103.031163] Key type lgssc registered [ 104.951633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 123.074786] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 165.285419] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 180.157917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 180.187435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.554056] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 181.602411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 181.747524] Lustre: lustre-MDT0000: new disk, initializing [ 181.869529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.880147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 187.240432] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.131746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.288906] Lustre: 6526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.323415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 202.327715] Lustre: Skipped 1 previous similar message [ 202.431718] Lustre: lustre-MDT0001: new disk, initializing [ 202.506109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.527558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 202.543356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 207.421222] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.584826] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 225.154245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.666343] Lustre: lustre-OST0000: new disk, initializing [ 225.670795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 225.679957] Lustre: 8479:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.848337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.969966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 232.977177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 233.049496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 234.021884] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.707110] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.799332] Lustre: lustre-OST0001: new disk, initializing [ 248.803433] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 248.806173] Lustre: 9552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.843081] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 255.805534] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.522971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 256.526546] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 256.573631] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 268.177413] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 276.179549] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 282.591807] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing check_logdir /tmp/testlogs/ [ 286.989534] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing yml_node [ 290.265198] Lustre: DEBUG MARKER: Client: 2.17.56.3 [ 292.547528] Lustre: DEBUG MARKER: MDS: 2.17.56.3 [ 294.882328] Lustre: DEBUG MARKER: OSS: 2.17.56.3 [ 296.549647] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 7 04:02:30 EDT 2026 [ 316.135962] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 317.705361] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 333.282872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.284851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.306267] Lustre: Skipped 1 previous similar message [ 338.400493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.404647] Lustre: Skipped 6 previous similar messages [ 339.626634] Lustre: server umount lustre-MDT0000 complete [ 343.127297] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786089798 with bad export cookie 9381631952451677073 [ 343.133209] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.139208] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 343.521035] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.522754] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.545075] Lustre: Skipped 3 previous similar messages [ 343.562635] Lustre: Skipped 1 previous similar message [ 348.640200] LustreError: 6533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.640352] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.663162] LustreError: 6533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 348.706451] Lustre: Skipped 1 previous similar message [ 349.544667] Lustre: server umount lustre-MDT0001 complete [ 353.691690] Lustre: server umount lustre-OST0000 complete [ 358.089630] Lustre: server umount lustre-OST0001 complete [ 365.411885] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 374.081772] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 411.823148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.200264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 412.261182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 412.377416] Lustre: lustre-MDT0000: new disk, initializing [ 412.446988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.470851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 418.572477] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 430.747207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.863148] Lustre: 15106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 430.900570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 430.903563] Lustre: Skipped 1 previous similar message [ 430.956625] Lustre: lustre-MDT0001: new disk, initializing [ 431.015988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 431.056246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 431.063754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 436.305811] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 446.049321] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 453.174304] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 460.943566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 461.373105] Lustre: lustre-OST0000: new disk, initializing [ 461.377060] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 461.382977] Lustre: 17077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 461.495648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 463.285382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 463.306498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 463.453024] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 468.623446] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 478.871396] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 485.899821] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 486.037301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 491.984619] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 492.234495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 499.168410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 499.171754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.190242] LustreError: Skipped 1 previous similar message [ 499.191728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 499.224756] Lustre: Skipped 1 previous similar message [ 500.875532] Lustre: server umount lustre-OST0000 complete [ 507.872897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 507.878868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.890307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 507.899314] Lustre: Skipped 1 previous similar message [ 519.135598] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 519.387573] Lustre: server umount lustre-MDT0000 complete [ 519.660690] LustreError: 15117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.686438] LustreError: 15117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 523.231958] LustreError: 15098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786089978 with bad export cookie 9381631952451678606 [ 523.236954] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 523.241323] LustreError: 15098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 523.633613] Lustre: server umount lustre-MDT0001 complete [ 530.080283] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:06:23 (1786089983) [ 547.163939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.506443] LustreError: 19951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.613912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.640455] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 552.933837] LustreError: 19952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.028896] LustreError: 19951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.524525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.159490] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 573.110418] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 578.750672] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 581.301596] Lustre: Failing over lustre-MDT0000 [ 581.604286] Lustre: server umount lustre-MDT0000 complete [ 581.605955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 581.615129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.625517] Lustre: Skipped 1 previous similar message [ 581.635887] LustreError: 19951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.984647] Lustre: 3645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786090037/real 1786090037] req@ffffa00bb2340a80 x1872850642431232/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786090053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 598.023125] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 598.053237] LustreError: 19946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 598.058752] LustreError: 19946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 601.778469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.227516] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8232408e03563714 [ 608.238688] Lustre: MGC192.168.201.106@tcp: Connection restored to 0@lo (at 0@lo) [ 608.463977] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 612.884505] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 613.876389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 613.878500] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 613.934646] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 622.367566] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 628.204678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.207465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.211385] Lustre: Skipped 2 previous similar messages [ 628.224253] Lustre: Skipped 5 previous similar messages [ 632.693798] Lustre: server umount lustre-MDT0000 complete [ 635.389983] LustreError: 19951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.415429] LustreError: 19951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 636.125372] LustreError: 19934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786090091 with bad export cookie 9381631952451680020 [ 636.130113] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.132576] LustreError: 19934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 636.427454] Lustre: server umount lustre-MDT0001 complete [ 643.991209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.411590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 644.415941] Lustre: Skipped 2 previous similar messages [ 649.263059] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 658.970359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.428931] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 673.472564] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 680.948651] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 683.576048] Lustre: Failing over lustre-MDT0000 [ 683.594776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 683.603369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 683.606918] Lustre: Skipped 1 previous similar message [ 683.631952] Lustre: Skipped 2 previous similar messages [ 684.172342] Lustre: server umount lustre-MDT0000 complete [ 700.393119] LustreError: 23287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.420197] LustreError: 23287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 700.880660] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786090139/real 1786090139] req@ffffa00b847e1880 x1872850642460544/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786090155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 700.922486] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.222479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.141796] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8232408e03563abe [ 711.154858] Lustre: MGC192.168.201.106@tcp: Connection restored to 0@lo (at 0@lo) [ 711.172571] Lustre: Skipped 1 previous similar message [ 711.509268] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 716.770500] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 716.781117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 716.831610] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 717.270146] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 728.207799] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 734.215305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 734.242354] Lustre: Skipped 1 previous similar message [ 740.726615] Lustre: server umount lustre-MDT0000 complete [ 745.037162] LustreError: 23274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786090200 with bad export cookie 9381631952451680958 [ 745.054934] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 745.056135] LustreError: 23274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 745.460455] Lustre: server umount lustre-MDT0001 complete [ 754.990403] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:10:08 (1786090208) [ 774.837776] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 787.447439] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 834.902799] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 847.299105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.571479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 847.613912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 847.731825] Lustre: lustre-MDT0000: new disk, initializing [ 847.874425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 847.880636] Lustre: Skipped 2 previous similar messages [ 847.898785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 853.539641] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 866.553593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.717287] Lustre: 29671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 867.131666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 872.053181] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 883.315837] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 892.670811] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 901.234302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 901.474972] Lustre: 31641:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 902.802648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 902.964821] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 902.971912] Lustre: Skipped 1 previous similar message [ 903.036801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 903.072633] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 911.183835] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 923.568755] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 929.563655] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 954.337975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 954.340623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.351812] LustreError: Skipped 1 previous similar message [ 954.354607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 954.368613] Lustre: Skipped 1 previous similar message [ 954.379821] Lustre: Skipped 2 previous similar messages [ 956.700070] Lustre: server umount lustre-OST0000 complete [ 964.577801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 967.271416] Lustre: server umount lustre-MDT0000 complete [ 969.713809] LustreError: 29677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.735595] LustreError: 29677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 972.015638] LustreError: 29661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786090427 with bad export cookie 9381631952451681357 [ 972.018209] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 972.032709] LustreError: 29661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 972.409967] Lustre: server umount lustre-MDT0001 complete [ 986.805400] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 990.028960] Key type lgssc unregistered [ 990.392951] LNet: 33636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.403364] LNetError: 33636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.428956] LNet: Removed LNI 192.168.201.106@tcp [ 991.561172] Key type .llcrypt unregistered [ 991.564670] Key type ._llcrypt unregistered [ 1015.892082] Key type ._llcrypt registered [ 1015.894292] Key type .llcrypt registered [ 1015.994135] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1017.246545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1017.301749] alg: No test for adler32 (adler32-zlib) [ 1018.321706] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 1018.534849] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1020.175227] Key type lgssc registered [ 1021.395802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1031.119465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1031.135765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.668772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1037.300101] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1042.931645] LustreError: 35048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.946340] LustreError: 35048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1047.001727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.381217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1052.375895] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1061.113891] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1068.025212] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1074.730846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1075.302539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1078.418626] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1081.599557] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1089.917227] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1092.723359] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1110.169433] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 04:16:03 (1786090563) [ 1112.561825] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1114.647226] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1122.203335] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 04:16:15 (1786090575) [ 1131.950890] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 04:16:25 (1786090585) [ 1133.180833] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1142.018987] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1142.022555] Lustre: Skipped 2 previous similar messages [ 1143.866394] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 04:16:37 (1786090597) [ 1147.643774] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1147.646585] Lustre: Skipped 3 previous similar messages [ 1168.335273] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1168.342306] Lustre: Skipped 6 previous similar messages [ 1171.689409] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 04:17:05 (1786090625) [ 1174.598888] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1178.575661] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1188.166752] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 04:17:22 (1786090642) [ 1190.407402] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1190.414136] Lustre: Skipped 3 previous similar messages [ 1208.398803] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 04:17:42 (1786090662) [ 1231.330694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.333244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1231.341747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1231.375283] Lustre: Skipped 1 previous similar message [ 1236.458612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1236.474684] Lustre: Skipped 2 previous similar messages [ 1241.569250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.574852] Lustre: Skipped 1 previous similar message [ 1244.130051] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1244.349897] Lustre: server umount lustre-OST0000 complete [ 1251.813632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.816878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1251.822203] Lustre: Skipped 1 previous similar message [ 1251.837680] Lustre: Skipped 1 previous similar message [ 1255.226069] Lustre: server umount lustre-MDT0000 complete [ 1256.936896] LustreError: 35045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.964783] LustreError: 35045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1260.533933] LustreError: 40078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786090715 with bad export cookie 10741064544561175694 [ 1260.552333] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1260.554015] LustreError: 40078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1261.294377] Lustre: server umount lustre-MDT0001 complete [ 1276.686785] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1279.619095] Key type lgssc unregistered [ 1279.969718] LNet: 42908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.984352] LNetError: 42908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.020453] LNet: Removed LNI 192.168.201.106@tcp [ 1280.858695] Key type .llcrypt unregistered [ 1280.864227] Key type ._llcrypt unregistered [ 1301.520140] Key type ._llcrypt registered [ 1301.525708] Key type .llcrypt registered [ 1301.707597] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 1318.665112] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1320.372099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1320.421076] alg: No test for adler32 (adler32-zlib) [ 1321.597581] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 1322.118903] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1323.831241] Key type lgssc registered [ 1324.944579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1364.840826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1364.898451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.193485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1366.258855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1366.457315] Lustre: lustre-MDT0000: new disk, initializing [ 1366.626903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.662439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1371.702743] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1383.472059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.661354] Lustre: 46065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1383.762496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1383.770089] Lustre: Skipped 1 previous similar message [ 1383.971349] Lustre: lustre-MDT0001: new disk, initializing [ 1384.083256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1384.165349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1384.179105] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1388.898399] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1398.125888] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1404.906629] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1411.914121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1412.416776] Lustre: lustre-OST0000: new disk, initializing [ 1412.428764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1412.436561] Lustre: 48031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1412.594774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.610351] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1422.428750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1422.436805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1422.560384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1429.532469] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1435.031217] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1435.241771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1440.815294] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1441.035414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1443.299600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1443.306227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.306503] LustreError: Skipped 1 previous similar message [ 1443.307930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1443.315164] Lustre: Skipped 1 previous similar message [ 1443.332187] Lustre: Skipped 1 previous similar message [ 1448.418943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1448.427382] Lustre: Skipped 1 previous similar message [ 1449.456347] Lustre: server umount lustre-OST0000 complete [ 1453.538202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.555233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1453.561403] Lustre: Skipped 1 previous similar message [ 1458.656722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1458.668735] Lustre: Skipped 1 previous similar message [ 1463.777064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1463.786831] Lustre: Skipped 1 previous similar message [ 1467.359391] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1467.703101] Lustre: server umount lustre-MDT0000 complete [ 1471.659237] LustreError: 46056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786090926 with bad export cookie 10326750657863510844 [ 1471.666893] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1471.673950] LustreError: 46056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1471.978641] Lustre: server umount lustre-MDT0001 complete [ 1482.082532] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 04:22:15 (1786090935) [ 1489.348915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.850374] LustreError: 50494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.871469] LustreError: 50494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1489.950823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.483530] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1495.010811] LustreError: 50495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1499.105350] LustreError: 50494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1502.693660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1503.144809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1508.946366] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1517.592971] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1523.940049] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1530.132238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1530.554837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.577755] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1544.206752] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1553.236815] LustreError: 53020:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1554.228308] Lustre: Modifying parameter general.timeout=1 in log params [ 1555.251454] Lustre: Modifying parameter general.timeout=2 in log params [ 1557.517317] Lustre: Modifying parameter general.timeout=5 in log params [ 1557.521307] Lustre: Skipped 2 previous similar messages [ 1561.554746] Lustre: Modifying parameter general.timeout=10 in log params [ 1561.558804] Lustre: Skipped 4 previous similar messages [ 1570.335834] Lustre: Modifying parameter general.timeout=21 in log params [ 1570.341483] Lustre: Skipped 10 previous similar messages [ 1586.680552] Lustre: Modifying parameter general.timeout=40 in log params [ 1586.686884] Lustre: Skipped 18 previous similar messages [ 1619.244180] Lustre: Modifying parameter general.timeout=80 in log params [ 1619.248354] Lustre: Skipped 39 previous similar messages [ 1680.090977] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 04:25:33 (1786091133) [ 1683.360957] Lustre: 59964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1684.394073] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1684.401592] Lustre: Skipped 52 previous similar messages [ 1685.399759] Lustre: 60063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1687.496236] Lustre: 60161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1691.780961] Lustre: 60374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1691.796938] Lustre: 60374:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1725.294913] Lustre: 61845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1725.313239] Lustre: 61845:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1725.367123] LustreError: 61845:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1730.838755] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 04:26:24 (1786091184) [ 1743.852879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1743.862060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.875970] Lustre: Skipped 1 previous similar message [ 1743.878714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1750.271126] Lustre: server umount lustre-MDT0000 complete [ 1750.507703] LustreError: 50495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1750.529370] LustreError: 50495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1754.554948] LustreError: 55251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786091209 with bad export cookie 10326750657863511726 [ 1754.561132] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1754.568928] LustreError: 55251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1755.028331] Lustre: server umount lustre-MDT0001 complete [ 1769.427674] Lustre: server umount lustre-OST0000 complete [ 1803.174528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.660755] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1804.019854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1809.199365] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1819.917366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1820.132343] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1820.190940] Lustre: 64379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1827.306134] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1835.894976] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1842.641868] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1850.822338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1851.091138] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1851.598614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1851.611128] Lustre: Skipped 1 previous similar message [ 1860.747259] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1869.411825] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1871.611960] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1871.621093] Lustre: Skipped 20 previous similar messages [ 1871.722192] Lustre: 66946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1885.154968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1885.164892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.176793] Lustre: Skipped 2 previous similar messages [ 1885.184461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1885.188723] Lustre: Skipped 3 previous similar messages [ 1888.063366] Lustre: server umount lustre-OST0000 complete [ 1892.321164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1892.337175] LustreError: Skipped 1 previous similar message [ 1892.342272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.365231] Lustre: Skipped 1 previous similar message [ 1906.143198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1906.362031] Lustre: server umount lustre-MDT0000 complete [ 1910.245705] LustreError: 64386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.272929] LustreError: 64386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1911.288109] LustreError: 64373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786091366 with bad export cookie 10326750657863514827 [ 1911.299074] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1911.301859] LustreError: 64373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1911.731685] Lustre: server umount lustre-MDT0001 complete [ 1923.655113] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1926.608373] Key type lgssc unregistered [ 1927.002751] LNet: 68161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.014663] LNetError: 68161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1927.044809] LNet: Removed LNI 192.168.201.106@tcp [ 1928.066229] Key type .llcrypt unregistered [ 1928.068288] Key type ._llcrypt unregistered [ 1954.001961] Key type ._llcrypt registered [ 1954.004404] Key type .llcrypt registered [ 1954.160916] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1955.191505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.409417] alg: No test for adler32 (adler32-zlib) [ 1956.507275] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 1956.704947] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1958.335219] Key type lgssc registered [ 1959.359314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1967.377345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1967.397421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.995163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1974.005277] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1979.363884] LustreError: 69578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.386972] LustreError: 69578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1983.939501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1984.848340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1989.517566] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1997.639908] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2005.034658] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2015.179609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2016.065233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.459148] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2030.823978] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2050.302821] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2065.377142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2065.383556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2065.404343] Lustre: Skipped 2 previous similar messages [ 2065.423278] Lustre: Skipped 2 previous similar messages [ 2068.009692] Lustre: server umount lustre-MDT0000 complete [ 2070.499634] LustreError: 69572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2070.517750] LustreError: 69572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2071.833182] LustreError: 69559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786091526 with bad export cookie 6259295338957042587 [ 2071.861963] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2072.252430] Lustre: server umount lustre-MDT0001 complete [ 2087.082378] Lustre: server umount lustre-OST0000 complete [ 2117.945677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.613324] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2118.683781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2119.246642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2124.831858] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2133.971727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.146922] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2134.197446] Lustre: 74468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2134.587117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2139.133961] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2147.074296] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2153.378180] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2160.632854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2160.928374] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2161.361225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2168.411537] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2177.300767] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2179.173820] Lustre: 77044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2181.645309] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2190.308821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2190.326902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.354258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2190.816204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2194.765853] Lustre: server umount lustre-OST0000 complete [ 2201.056325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.061628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2201.078467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2201.078476] Lustre: Skipped 1 previous similar message [ 2201.078829] Lustre: Skipped 2 previous similar messages [ 2206.180926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2206.190184] Lustre: Skipped 2 previous similar messages [ 2211.313131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2211.320460] Lustre: Skipped 1 previous similar message [ 2213.343185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2213.674821] Lustre: server umount lustre-MDT0000 complete [ 2216.417264] LustreError: 74480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.442836] LustreError: 74480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2217.531431] LustreError: 74461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786091672 with bad export cookie 6259295338957043434 [ 2217.539673] LustreError: 74461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2217.543522] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2218.101403] Lustre: server umount lustre-MDT0001 complete [ 2233.878569] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2236.937579] Key type lgssc unregistered [ 2237.328094] LNet: 78261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2237.337127] LNetError: 78261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.376146] LNet: Removed LNI 192.168.201.106@tcp [ 2239.435118] Key type .llcrypt unregistered [ 2239.439411] Key type ._llcrypt unregistered [ 2271.644679] Key type ._llcrypt registered [ 2271.649913] Key type .llcrypt registered [ 2271.828914] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2273.766737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2274.028983] alg: No test for adler32 (adler32-zlib) [ 2275.355819] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 2275.656785] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2277.327208] Key type lgssc registered [ 2278.793908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2287.111266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2287.139449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.692965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2293.780430] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2298.850079] LustreError: 79676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.875346] LustreError: 79676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2303.099459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.777991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2309.077453] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2317.015816] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2324.209818] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2331.394443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2332.157284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2340.738684] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2348.929318] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2354.707636] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2355.619526] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2357.236592] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2357.243308] Lustre: Skipped 1 previous similar message [ 2359.832366] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2359.836801] Lustre: Skipped 2 previous similar messages [ 2363.951721] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2363.956381] Lustre: Skipped 4 previous similar messages [ 2372.243799] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2372.257253] Lustre: Skipped 8 previous similar messages [ 2388.296777] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2388.301994] Lustre: Skipped 17 previous similar messages [ 2420.457428] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2420.467244] Lustre: Skipped 34 previous similar messages [ 2462.287110] Lustre: 87778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2465.309204] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 04:38:39 (1786091919) [ 2467.123907] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2469.858679] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 04:38:43 (1786091923) [ 2476.519027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2476.525921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.549931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2481.633921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2481.636399] Lustre: Skipped 1 previous similar message [ 2486.753141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.764175] Lustre: Skipped 1 previous similar message [ 2489.311528] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2489.475106] Lustre: server umount lustre-OST0000 complete [ 2496.992893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.003563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2497.007810] Lustre: Skipped 2 previous similar messages [ 2497.028052] Lustre: Skipped 1 previous similar message [ 2499.934258] Lustre: server umount lustre-MDT0000 complete [ 2502.112192] LustreError: 79676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.125921] LustreError: 79676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2503.123637] LustreError: 79657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786091958 with bad export cookie 4998219930937915969 [ 2503.132076] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2503.132224] LustreError: 79657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2503.498546] Lustre: server umount lustre-MDT0001 complete [ 2514.365434] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2517.054184] Key type lgssc unregistered [ 2517.293460] LNet: 89028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2517.297634] LNetError: 89028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2518.311428] LNet: Removed LNI 192.168.201.106@tcp [ 2519.182639] Key type .llcrypt unregistered [ 2519.184423] Key type ._llcrypt unregistered [ 2524.707626] Key type ._llcrypt registered [ 2524.710074] Key type .llcrypt registered [ 2524.885416] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2530.696609] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules [ 2532.443931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2532.695469] alg: No test for adler32 (adler32-zlib) [ 2534.156209] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 2534.649945] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2536.561074] Key type lgssc registered [ 2537.926367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2539.435939] LustreError: 90264:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2559.319650] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2561.313277] LustreError: 90264:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2569.271984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2569.284709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2570.863749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.492928] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2580.453555] LustreError: 91255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.480047] LustreError: 91255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2585.570032] LustreError: 91256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.636405] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 04:40:41 (1786092041) [ 2590.690218] LustreError: 91255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.792187] LustreError: 91287:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2592.806925] LustreError: 91287:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2592.823275] LustreError: 91287:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2593.323066] Lustre: server umount lustre-MDT0000 complete [ 2606.406113] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2609.241605] Key type lgssc unregistered [ 2609.481967] LNet: 92782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2609.486760] LNetError: 92782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2609.497429] LNet: Removed LNI 192.168.201.106@tcp [ 2610.358343] Key type .llcrypt unregistered [ 2610.363488] Key type ._llcrypt unregistered [ 2631.667467] Key type ._llcrypt registered [ 2631.671308] Key type .llcrypt registered [ 2631.774240] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2632.864331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2632.901576] alg: No test for adler32 (adler32-zlib) [ 2634.092870] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 2634.387360] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2636.071242] Key type lgssc registered [ 2637.435583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2646.387045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2646.397440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2647.926089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2652.468718] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2657.252516] LustreError: 94003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2657.274993] LustreError: 94003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2661.929445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.569971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2668.218902] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2676.937682] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2682.273862] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2688.539358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2689.088495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.758082] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2702.685891] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2751.684378] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 04:43:25 (1786092205) [ 2753.242221] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2755.225487] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 04:43:29 (1786092209) [ 2759.648291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2759.656463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.671417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.744978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.747399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.763204] Lustre: Skipped 1 previous similar message [ 2763.771241] Lustre: Skipped 1 previous similar message [ 2766.002608] Lustre: server umount lustre-MDT0000 complete [ 2768.867130] LustreError: 93999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.893804] LustreError: 93999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2769.812319] LustreError: 93983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786092224 with bad export cookie 18389541678531460657 [ 2769.838735] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2770.259520] Lustre: server umount lustre-MDT0001 complete [ 2784.866428] Lustre: server umount lustre-OST0000 complete [ 2799.508829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.038080] LustreError: 98277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.285068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.707410] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2805.735430] LustreError: 98278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.860641] LustreError: 98277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.708957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.434615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2822.574100] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2831.691704] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2838.005848] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2852.592722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2852.876574] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2852.880907] LustreError: 98265:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2852.937804] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2852.962823] LustreError: 100363:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2852.965493] LustreError: 100363:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2852.991092] LustreError: 100363:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2853.114321] Lustre: server umount lustre-OST0000 complete [ 2853.123410] LustreError: 100363:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2861.971508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2862.231211] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2862.236187] LustreError: 98265:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2862.259462] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2862.266889] LustreError: 100831:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2862.271569] LustreError: 100831:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2862.275428] LustreError: 100831:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2862.340232] Lustre: server umount lustre-OST0000 complete [ 2862.344376] LustreError: 100831:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2870.629737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2870.775118] LustreError: 101347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2870.827179] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2870.831329] Lustre: Found index 0 for lustre-OST0000, updating log [ 2870.836594] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2870.910018] Lustre: lustre-OST0000: new disk, initializing [ 2870.913493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2870.922176] Lustre: 101364:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2871.173311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.062996] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2880.542677] Lustre: Failing over lustre-OST0000 [ 2880.726279] Lustre: server umount lustre-OST0000 complete [ 2885.795448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.801683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2885.808777] Lustre: Skipped 1 previous similar message [ 2891.233149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.242504] Lustre: Skipped 2 previous similar messages [ 2894.427543] Lustre: server umount lustre-MDT0000 complete [ 2896.352671] LustreError: 99009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2896.381351] LustreError: 99009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2898.524815] LustreError: 100383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786092353 with bad export cookie 18389541678531463401 [ 2898.537313] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2898.538508] LustreError: 100383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2899.066515] Lustre: server umount lustre-MDT0001 complete [ 2907.546463] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 04:46:01 (1786092361) [ 2924.964613] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2935.712062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.006675] LustreError: 103710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.146676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.704175] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2949.763880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.420194] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2958.678174] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2965.530694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2966.436835] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2973.807361] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2975.662429] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2975.671513] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2975.704400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2980.850089] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2980.884158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2982.732392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2982.878362] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2982.958598] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2982.981666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2982.991664] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2982.994983] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2983.018242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2983.046408] Lustre: lustre-OST0001: new disk, initializing [ 2983.052575] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2983.064092] Lustre: 106029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2983.282301] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2983.290212] Lustre: Skipped 2 previous similar messages [ 2989.804512] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2990.679430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2990.688956] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2990.824877] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2997.414534] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3002.895394] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3006.946067] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3006.956514] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.972751] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3011.555973] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.563015] Lustre: Skipped 1 previous similar message [ 3012.947957] LustreError: 105057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3012.980445] LustreError: 105057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3013.023485] Lustre: server umount lustre-MDT0001 complete [ 3024.574395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.749301] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3024.759676] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3024.887068] Lustre: 103700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3026.738853] LustreError: 107570:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3026.749759] LustreError: 107570:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3026.764726] LustreError: 103702:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3026.868536] LustreError: 107578:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.106@tcp: cfg command failed: rc = -17 [ 3026.883671] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3026.883671] [ 3030.856397] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3031.548344] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3039.885465] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 04:48:13 (1786092493) [ 3056.185905] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3074.393918] Lustre: 109643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3095.466024] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3098.175481] Lustre: 110777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3178.983126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3178.987471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3179.012799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3179.019139] Lustre: Skipped 4 previous similar messages [ 3184.961323] Lustre: server umount lustre-MDT0000 complete [ 3185.124262] LustreError: 105057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.147566] LustreError: 105057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3189.363862] LustreError: 103692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786092644 with bad export cookie 18389541678531464402 [ 3189.373519] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3189.377901] LustreError: 103692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3189.770682] Lustre: server umount lustre-MDT0001 complete [ 3204.292114] Lustre: server umount lustre-OST0000 complete [ 3206.240142] Lustre: 93316:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786092645/real 1786092645] req@ffffa00a94106680 x1872853298671744/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786092661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3206.262951] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.277391] Lustre: Skipped 3 previous similar messages [ 3207.924399] Lustre: server umount lustre-OST0001 complete [ 3213.333206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3222.341980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3230.974248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3239.542678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3256.507516] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3267.036182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3267.066856] Lustre: lustre-MDT0000: reset Object Index mappings [ 3267.799365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.803541] Lustre: Skipped 1 previous similar message [ 3272.562597] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3281.188465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3281.202187] Lustre: lustre-MDT0001: reset Object Index mappings [ 3285.684884] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3288.634234] Lustre: 115877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3294.639677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3300.957806] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3306.547293] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3309.127075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3312.897879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3312.928252] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3312.974135] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3317.822408] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3328.127368] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3332.495288] Lustre: 117755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3387.099858] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 04:54:00 (1786092840) [ 3414.504174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3414.521924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3414.552898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3414.569733] Lustre: Skipped 4 previous similar messages [ 3417.638061] Lustre: server umount lustre-MDT0000 complete [ 3422.389726] LustreError: 114711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786092877 with bad export cookie 18389541678531511470 [ 3422.409463] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3429.150255] Lustre: server umount lustre-MDT0001 complete [ 3439.795658] Lustre: server umount lustre-OST0000 complete [ 3450.208775] Lustre: server umount lustre-OST0001 complete [ 3459.486851] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 3471.974483] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3509.786311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.027160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3510.060195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3510.128751] Lustre: lustre-MDT0000: new disk, initializing [ 3510.194191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.197376] Lustre: Skipped 3 previous similar messages [ 3510.215772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3514.582732] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3521.585461] Lustre: server umount lustre-MDT0000 complete [ 3531.875562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3532.203864] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3532.214293] Lustre: 122026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3532.228589] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3537.206790] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3542.156311] Lustre: server umount lustre-MDT0000 complete [ 3550.680152] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 04:56:44 (1786093004) [ 3552.376612] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3554.227465] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 04:56:48 (1786093008) [ 3574.601047] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3586.017412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3591.606909] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3600.524392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3600.681585] Lustre: 124077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3600.765974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3600.770257] Lustre: Skipped 1 previous similar message [ 3600.877368] Lustre: lustre-MDT0001: new disk, initializing [ 3601.144341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3601.153246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3607.991476] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3614.155659] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3622.042580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3622.347258] Lustre: lustre-OST0000: new disk, initializing [ 3622.350359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3622.356279] Lustre: 125658:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3624.423382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3624.449951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3624.602206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3630.502750] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3645.678833] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3645.847451] Lustre: lustre-OST0001: new disk, initializing [ 3645.853620] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3645.867864] Lustre: 126528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3647.145138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3647.162777] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3647.358157] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3652.918516] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3663.050544] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3667.015786] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4048.156921] Lustre: 132950:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4048.164178] Lustre: 132950:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4048.169704] Lustre: 132950:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4048.178567] Lustre: 132950:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4048.186601] Lustre: 132950:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4048.195300] Lustre: 132950:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4299.199303] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:09:11 (1786093751) [ 4317.664144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4317.669967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4317.693620] Lustre: Skipped 5 previous similar messages [ 4317.700668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4317.712858] Lustre: Skipped 8 previous similar messages [ 4321.786958] Lustre: server umount lustre-MDT0000 complete [ 4323.304870] LustreError: 127345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4323.328679] LustreError: 127345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 4325.520420] LustreError: 132962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786093780 with bad export cookie 18389541678531578964 [ 4325.537019] LustreError: 132962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4325.547590] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4328.417665] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.429798] Lustre: Skipped 3 previous similar messages [ 4328.436669] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4328.442145] Lustre: Skipped 5 previous similar messages [ 4332.068583] Lustre: server umount lustre-MDT0001 complete [ 4341.791419] Lustre: server umount lustre-OST0000 complete [ 4351.674187] Lustre: server umount lustre-OST0001 complete [ 4357.726339] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 4367.892798] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4403.863676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.122646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4404.190966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4404.274335] Lustre: lustre-MDT0000: new disk, initializing [ 4404.359352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4404.370627] Lustre: Skipped 5 previous similar messages [ 4404.384601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4409.335837] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4419.830322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4420.003616] Lustre: 135998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4420.112822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4420.117855] Lustre: Skipped 1 previous similar message [ 4420.225416] Lustre: lustre-MDT0001: new disk, initializing [ 4420.352330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4420.363503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4425.048241] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4433.942405] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4439.303789] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4445.554086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4445.829676] Lustre: lustre-OST0000: new disk, initializing [ 4445.833112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4445.838436] Lustre: 137964:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4445.919418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4445.926871] Lustre: Skipped 1 previous similar message [ 4447.161086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4447.168858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4447.261938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4452.713753] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4463.284833] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4474.682472] Lustre: lustre-MDD0000: changelog on [ 4477.314393] Lustre: lustre-MDD0001: changelog on [ 6242.034549] Lustre: lustre-MDD0001: changelog off [ 6243.930301] Lustre: 138048:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6244.798944] Lustre: lustre-MDD0000: changelog off [ 6254.561233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6254.567739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.585702] Lustre: Skipped 1 previous similar message [ 6254.592979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6254.597104] Lustre: Skipped 3 previous similar messages [ 6259.692555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6259.699144] Lustre: Skipped 1 previous similar message [ 6264.287294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6264.514649] Lustre: server umount lustre-OST0000 complete [ 6268.897978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6268.902536] LustreError: Skipped 1 previous similar message [ 6268.912283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6268.929138] Lustre: Skipped 1 previous similar message [ 6268.935785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6274.145316] Lustre: server umount lustre-MDT0000 complete [ 6275.040376] LustreError: 136004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6275.066044] LustreError: 136004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6278.001625] LustreError: 135988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786095733 with bad export cookie 18389541678532359100 [ 6278.006735] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6278.014793] LustreError: 135988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6278.284047] Lustre: server umount lustre-MDT0001 complete [ 6289.913242] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 6292.698629] Key type lgssc unregistered [ 6292.947780] LNet: 147080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6292.953727] LNetError: 147080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6292.970273] LNet: Removed LNI 192.168.201.106@tcp [ 6294.002167] Key type .llcrypt unregistered [ 6294.005244] Key type ._llcrypt unregistered [ 6315.446414] Key type ._llcrypt registered [ 6315.448397] Key type .llcrypt registered [ 6315.532863] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 6330.052669] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6331.432611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6331.450661] alg: No test for adler32 (adler32-zlib) [ 6332.460806] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 6332.726752] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 6334.367922] Key type lgssc registered [ 6335.258936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6367.610233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6367.630713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6368.911474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6368.944869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6369.029971] Lustre: lustre-MDT0000: new disk, initializing [ 6369.133057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6369.151762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6373.757085] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6386.409254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6386.608565] Lustre: 150284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6386.695521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6386.700606] Lustre: Skipped 1 previous similar message [ 6386.822839] Lustre: lustre-MDT0001: new disk, initializing [ 6386.985219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6387.037596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6387.043468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6392.797913] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6402.544297] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6407.299203] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6413.691211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6413.995714] Lustre: lustre-OST0000: new disk, initializing [ 6414.004779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6414.014608] Lustre: 152253:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6414.194328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6420.795386] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6423.615946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6423.626902] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6423.782967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6429.334968] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6435.845585] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_module obdecho/obdecho [ 6437.734202] LustreError: 153256:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6437.743748] LustreError: 153256:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6439.453214] LustreError: 153352:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6454.752763] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6454.757664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6454.787713] Lustre: Skipped 2 previous similar messages [ 6454.807424] Lustre: Skipped 2 previous similar messages [ 6459.894589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6459.899686] Lustre: Skipped 2 previous similar messages [ 6460.843766] Lustre: server umount lustre-MDT0000 complete [ 6465.773320] LustreError: 150278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786095920 with bad export cookie 8202637663066334314 [ 6465.780237] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6465.803507] LustreError: 150278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6466.460630] Lustre: server umount lustre-MDT0001 complete [ 6481.494410] Lustre: server umount lustre-OST0000 complete [ 6502.581804] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 6511.994590] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6547.748450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6547.968939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6548.020990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6548.131351] Lustre: lustre-MDT0000: new disk, initializing [ 6548.199580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6548.224650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6552.985644] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6562.529424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6562.627460] Lustre: 157065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6562.691627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6562.699964] Lustre: Skipped 1 previous similar message [ 6562.704860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6562.783456] Lustre: lustre-MDT0001: new disk, initializing [ 6562.878302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6562.931645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6562.943904] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6567.465656] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6576.777215] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6583.050047] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6590.132885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6590.499892] Lustre: lustre-OST0000: new disk, initializing [ 6590.506297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6590.512949] Lustre: 159034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6590.617801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6597.028500] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6598.213681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6598.222623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6598.316854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6605.719197] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6609.304907] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6609.500304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6613.576403] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6613.724688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6619.110931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6619.113183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6619.126672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6619.133222] Lustre: Skipped 1 previous similar message [ 6619.158513] Lustre: Skipped 1 previous similar message [ 6621.771865] Lustre: server umount lustre-OST0000 complete [ 6629.344385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6629.349260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6629.386051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6629.395383] Lustre: Skipped 1 previous similar message [ 6631.795982] Lustre: server umount lustre-MDT0000 complete [ 6634.467400] LustreError: 157073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6634.481736] LustreError: 157073:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6635.671906] LustreError: 157058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786096090 with bad export cookie 8202637663066335273 [ 6635.682158] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6635.687926] LustreError: 157058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6636.103140] Lustre: server umount lustre-MDT0001 complete [ 6640.240486] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 05:48:14 (1786096094) [ 6642.523799] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6644.438523] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 05:48:18 (1786096098) [ 6652.593042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6653.355910] LustreError: 161448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.479639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6658.530466] LustreError: 161449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6659.344577] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6663.649319] LustreError: 161448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6668.770642] LustreError: 161449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6669.276867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6675.138663] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6683.606917] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6689.418418] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6696.080694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6696.479798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6696.487775] Lustre: Skipped 1 previous similar message [ 6703.545484] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6710.719253] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6724.067908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6724.083303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6724.105636] Lustre: Skipped 2 previous similar messages [ 6728.885806] Lustre: server umount lustre-MDT0000 complete [ 6729.193742] LustreError: 162414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6729.207793] LustreError: 162414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6732.515773] LustreError: 161430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786096187 with bad export cookie 8202637663066336155 [ 6732.532216] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6734.306092] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6734.317146] Lustre: Skipped 1 previous similar message [ 6739.029294] Lustre: server umount lustre-MDT0001 complete [ 6745.994088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6759.321798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.533613] LustreError: 165291:0:(llog.c:1655:llog_backup()) MGC192.168.201.106@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6759.544173] Lustre: 165291:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.106@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6761.953186] LustreError: 165296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6761.985482] LustreError: 165296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6797.792243] LustreError: 165298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6797.815864] LustreError: 165298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 6820.319613] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6820.328272] Lustre: 165291:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.106@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6861.302599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6864.193031] LustreError: 165317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6864.201621] LustreError: 165317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 6865.525573] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6866.405212] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6873.440115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6882.933557] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6883.322245] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6890.704313] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6892.242965] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6897.147711] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6898.807554] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6915.225533] Lustre: server umount lustre-MDT0000 complete [ 6918.826957] LustreError: 161428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786096373 with bad export cookie 8202637663066342609 [ 6918.844420] LustreError: 161428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6918.853995] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6919.137471] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6919.236131] Lustre: server umount lustre-MDT0001 complete [ 6923.538178] Lustre: server umount lustre-OST0000 complete [ 6946.383756] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 6957.883948] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6994.745693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.071806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6995.094867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6995.159090] Lustre: lustre-MDT0000: new disk, initializing [ 6995.219987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6995.226697] Lustre: Skipped 1 previous similar message [ 6995.240781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7000.359228] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7011.636712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7011.768378] Lustre: 170391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7012.034654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7016.675409] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7025.710254] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7031.081247] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7038.695340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7038.921813] Lustre: lustre-OST0000: new disk, initializing [ 7038.923931] Lustre: Skipped 1 previous similar message [ 7038.928892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7038.934647] Lustre: Skipped 2 previous similar messages [ 7038.938135] Lustre: 172361:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7044.702992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7044.731267] Lustre: Skipped 1 previous similar message [ 7044.744602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7044.927773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7045.705679] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7055.751439] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7060.182896] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7060.347096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7064.686678] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7064.936703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7070.690210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7070.692083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7070.708056] LustreError: Skipped 1 previous similar message [ 7070.713626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7070.717708] Lustre: Skipped 2 previous similar messages [ 7070.736348] Lustre: Skipped 7 previous similar messages [ 7073.027340] Lustre: server umount lustre-OST0000 complete [ 7078.881263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7083.318840] Lustre: server umount lustre-MDT0000 complete [ 7086.054541] LustreError: 171361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7086.075359] LustreError: 171361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 7087.350302] LustreError: 170385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786096542 with bad export cookie 8202637663066359437 [ 7087.353191] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7087.359233] LustreError: 170385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7087.774699] Lustre: server umount lustre-MDT0001 complete [ 7094.764525] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 05:55:49 (1786096549) [ 7101.364272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7106.390684] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7114.984576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.165716] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7129.385395] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7136.714533] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7144.624096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7151.239438] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7158.161937] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7171.782707] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 05:57:05 (1786096625) [ 7177.705795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7177.709576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7177.738213] Lustre: Skipped 2 previous similar messages [ 7177.760116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7177.764053] Lustre: Skipped 3 previous similar messages [ 7183.276146] Lustre: server umount lustre-OST0000 complete [ 7187.427805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7187.432773] LustreError: Skipped 1 previous similar message [ 7193.765351] Lustre: server umount lustre-MDT0000 complete [ 7197.155100] LustreError: 174751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786096652 with bad export cookie 8202637663066360319 [ 7197.167131] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7197.167863] LustreError: 174751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7197.526436] Lustre: server umount lustre-MDT0001 complete [ 7207.817895] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 7210.925934] Key type lgssc unregistered [ 7211.155201] LNet: 178498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7211.163887] LNetError: 178498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7212.218614] LNet: Removed LNI 192.168.201.106@tcp [ 7213.491572] Key type .llcrypt unregistered [ 7213.493195] Key type ._llcrypt unregistered [ 7236.702385] Key type ._llcrypt registered [ 7236.707608] Key type .llcrypt registered [ 7236.816495] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7238.638775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7238.747925] alg: No test for adler32 (adler32-zlib) [ 7240.071850] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 7240.414108] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 7242.185715] Key type lgssc registered [ 7243.606554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7253.562115] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7253.594778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7269.215733] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7274.335464] LustreError: 179743:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.106@tcp: failed processing log, type 4: rc = -110 [ 7279.457207] LustreError: 179743:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7279.470770] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7279.475742] LustreError: MGC192.168.201.106@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7279.484668] LustreError: 179743:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7279.493393] LustreError: 179743:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7279.510183] LustreError: 179743:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7279.594972] Lustre: server umount lustre-OST0000 complete [ 7279.602117] LustreError: 179743:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7288.742439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7290.278559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7296.534909] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7300.596440] LustreError: 180242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7300.618361] LustreError: 180242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7305.702053] LustreError: 180243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7306.621743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7307.115112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7311.818249] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7321.913106] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7327.826719] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7334.359868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7334.571041] LustreError: 182059:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7334.581917] Lustre: 182059:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.106@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7334.706508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7340.235227] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7347.523537] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7351.266058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.271646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7351.280909] Lustre: Skipped 1 previous similar message [ 7351.293512] Lustre: Skipped 2 previous similar messages [ 7352.352611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7352.357315] Lustre: Skipped 2 previous similar messages [ 7355.767286] Lustre: server umount lustre-MDT0000 complete [ 7357.411236] LustreError: 180243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7357.435759] LustreError: 180243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7359.948177] LustreError: 182087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786096815 with bad export cookie 2501792744684967212 [ 7359.951193] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7359.961762] LustreError: 182087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7360.503349] LustreError: 180972:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001b24e5af x1872858128087040/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7361.001259] Lustre: server umount lustre-MDT0001 complete [ 7377.208932] Lustre: server umount lustre-OST0000 complete [ 7389.958491] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 7400.658494] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7439.446164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7439.755205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7439.800642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7439.934767] Lustre: lustre-MDT0000: new disk, initializing [ 7440.023283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7440.036406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7444.876801] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7455.486567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7455.619685] Lustre: 185598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7455.713326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7455.719838] Lustre: Skipped 1 previous similar message [ 7455.834396] Lustre: lustre-MDT0001: new disk, initializing [ 7455.969476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7456.006370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7456.017777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7460.691395] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7470.145719] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7475.702722] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7482.503517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7482.674742] Lustre: lustre-OST0000: new disk, initializing [ 7482.678265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7482.683534] Lustre: 187564:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7482.763994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7488.660852] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7489.110591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7489.116579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7489.270125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7498.161286] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7503.212074] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7503.403167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7508.209153] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7508.418503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7515.118697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7515.132555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7515.152985] Lustre: Skipped 1 previous similar message [ 7515.155715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7515.177381] LustreError: Skipped 1 previous similar message [ 7516.540996] Lustre: server umount lustre-OST0000 complete [ 7522.784288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7522.792782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7522.800959] Lustre: Skipped 1 previous similar message [ 7522.807788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7522.810685] Lustre: Skipped 1 previous similar message [ 7525.346475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7526.737987] Lustre: server umount lustre-MDT0000 complete [ 7530.473517] LustreError: 188599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7530.518683] LustreError: 188599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7530.719277] LustreError: 185592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786096985 with bad export cookie 2501792744684967926 [ 7530.743319] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7530.764663] LustreError: 185592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7531.100234] Lustre: server umount lustre-MDT0001 complete [ 7541.074062] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:03:14 (1786096994) [ 7559.561943] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 7562.566363] Key type lgssc unregistered [ 7562.886037] LNet: 190512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7562.904525] LNetError: 190512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7562.964956] LNet: Removed LNI 192.168.201.106@tcp [ 7564.086275] Key type .llcrypt unregistered [ 7564.088128] Key type ._llcrypt unregistered [ 7585.390562] Key type ._llcrypt registered [ 7585.392727] Key type .llcrypt registered [ 7585.497285] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7586.467775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7586.494314] alg: No test for adler32 (adler32-zlib) [ 7587.549838] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 7587.820098] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 7589.471291] Key type lgssc registered [ 7590.723264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7597.983222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7598.000150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7599.452697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7604.335334] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7609.825424] LustreError: 191731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7609.840288] LustreError: 191731:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7614.392286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7614.956920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7619.374719] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7626.606679] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7632.506196] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7641.570206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7641.861637] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7641.868755] LustreError: 193595:0:(llog.c:1696:llog_backup()) MGC192.168.201.106@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7641.877477] Lustre: 193595:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.106@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7642.066983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7649.177350] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7656.621524] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7659.490463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7659.499516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7659.520854] Lustre: Skipped 2 previous similar messages [ 7659.545468] Lustre: Skipped 2 previous similar messages [ 7660.586607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7665.642573] LustreError: 191732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7665.667933] LustreError: 191732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7666.961836] Lustre: server umount lustre-MDT0000 complete [ 7670.425137] LustreError: 191711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786097125 with bad export cookie 17749588747430977337 [ 7670.428794] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7670.447791] LustreError: 191711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7670.752263] LustreError: 191727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7670.754210] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7670.777734] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7670.782347] Lustre: Skipped 2 previous similar messages [ 7670.840994] LustreError: 192462:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007123aa97 x1872858491941248/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7671.018246] Lustre: server umount lustre-MDT0001 complete [ 7676.850179] Lustre: server umount lustre-OST0000 complete [ 7686.786608] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 7699.210864] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7744.786825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7745.135579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7745.242720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7745.449078] Lustre: lustre-MDT0000: new disk, initializing [ 7745.595852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7745.625208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7750.928847] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7763.400268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7763.535179] Lustre: 197124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7763.598593] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7763.600807] Lustre: Skipped 1 previous similar message [ 7763.605537] LustreError: 197907:0:(llog.c:1696:llog_backup()) MGC192.168.201.106@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7763.611869] LustreError: 197907:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7763.617167] Lustre: 197907:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.106@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7763.641554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7763.645227] Lustre: Skipped 1 previous similar message [ 7763.777555] Lustre: lustre-MDT0001: new disk, initializing [ 7763.935719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7764.003224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7764.020094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7768.689824] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7778.420497] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7784.384310] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7792.585892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7792.904836] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7792.908592] Lustre: Skipped 1 previous similar message [ 7792.911975] LustreError: 199070:0:(llog.c:1696:llog_backup()) MGC192.168.201.106@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7792.919385] LustreError: 199070:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7792.927154] Lustre: 199070:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.106@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7792.939183] Lustre: 199070:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7792.966082] Lustre: lustre-OST0000: new disk, initializing [ 7792.970978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7792.988813] Lustre: 199093:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7793.097516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7799.908602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7799.931318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7800.134554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7801.911465] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7814.786498] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7819.929843] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7820.166674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7824.424530] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7824.683205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7831.009717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7831.012704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7831.017463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7831.024989] LustreError: Skipped 1 previous similar message [ 7831.078365] Lustre: Skipped 1 previous similar message [ 7833.672439] Lustre: server umount lustre-OST0000 complete [ 7840.738084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7840.746554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7840.763119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7840.770402] Lustre: Skipped 1 previous similar message [ 7845.182553] Lustre: server umount lustre-MDT0000 complete [ 7846.368801] LustreError: 197135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7846.392708] LustreError: 197135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7850.709146] LustreError: 197920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786097305 with bad export cookie 17749588747430978058 [ 7850.713712] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7850.717778] LustreError: 197920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7851.297131] Lustre: server umount lustre-MDT0001 complete [ 7862.939640] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:08:35 (1786097315) [ 7882.877995] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7893.140675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.409906] LustreError: 202221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7893.480054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7897.212042] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7903.718072] LustreError: 202222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7903.730167] LustreError: 202222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7905.389036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7911.594436] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7914.909209] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7923.323675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7929.997384] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7938.979338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7939.299730] Lustre: lustre-OST0001: new disk, initializing [ 7939.308479] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7939.315731] Lustre: 204529:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7939.450177] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7939.452873] Lustre: Skipped 2 previous similar messages [ 7946.202175] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7946.897352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7946.913739] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7947.011624] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7956.821046] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7963.122152] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7964.894508] Lustre: DEBUG MARKER: ADD OST3 [ 7969.912859] loop: module loaded [ 7969.970438] loop0: detected capacity change from 0 to 204800000 [ 7970.223511] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7970.255394] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7970.275624] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7977.260216] loop0: detected capacity change from 0 to 204800000 [ 7977.695451] Lustre: DEBUG MARKER: STOP OST3 [ 7983.201353] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7983.420105] Lustre: lustre-OST0003: new disk, initializing [ 7983.423281] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7983.426927] Lustre: 206565:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7985.042676] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7985.157194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7985.163244] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7985.201142] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7986.419334] Lustre: Failing over lustre-OST0003 [ 7986.534391] Lustre: server umount lustre-OST0003 complete [ 7987.106145] LustreError: 206711:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7989.735644] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7990.128232] LustreError: 204814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.252954] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.254823] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7990.267977] Lustre: Skipped 1 previous similar message [ 7998.457623] Lustre: DEBUG MARKER: START OST3 again [ 8005.582672] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8005.860866] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8005.870981] Lustre: Skipped 1 previous similar message [ 8005.882394] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8007.711168] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8007.755771] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8007.756707] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8012.784412] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8037.346983] Lustre: Failing over lustre-OST0003 [ 8037.688374] Lustre: server umount lustre-OST0003 complete [ 8038.887427] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8038.905734] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8038.927595] Lustre: Skipped 2 previous similar messages [ 8038.934041] LustreError: 203709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8038.975293] LustreError: 203709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8057.313044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8057.314089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8057.327362] Lustre: Skipped 3 previous similar messages [ 8057.330088] Lustre: Skipped 4 previous similar messages [ 8062.948538] Lustre: server umount lustre-MDT0000 complete [ 8066.144141] LustreError: 202201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786097521 with bad export cookie 17749588747430978940 [ 8066.149883] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8066.159455] LustreError: 202201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8072.819128] Lustre: server umount lustre-MDT0001 complete [ 8076.699551] Lustre: server umount lustre-OST0000 complete [ 8080.218637] Lustre: server umount lustre-OST0001 complete [ 8086.919755] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8094.444389] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8127.198942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8127.449881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8127.475777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8127.579744] Lustre: lustre-MDT0000: new disk, initializing [ 8127.673853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8131.172666] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8143.192420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8143.382942] Lustre: 211993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8143.658682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8143.666256] Lustre: Skipped 1 previous similar message [ 8143.736628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8143.745279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8150.107815] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8162.733294] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8169.248282] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8176.932817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8177.267243] Lustre: lustre-OST0000: new disk, initializing [ 8177.274035] Lustre: Skipped 1 previous similar message [ 8177.291800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8177.299890] Lustre: Skipped 2 previous similar messages [ 8177.305642] Lustre: 213959:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8179.070537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8179.079144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8179.200224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8183.539285] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8192.081096] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8196.312065] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8196.479720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8201.072718] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8201.365598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8204.776439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8204.778052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8204.794165] Lustre: Skipped 2 previous similar messages [ 8204.797705] LustreError: Skipped 1 previous similar message [ 8204.807028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8204.825433] Lustre: Skipped 10 previous similar messages [ 8209.483094] Lustre: server umount lustre-OST0000 complete [ 8219.887917] Lustre: server umount lustre-MDT0000 complete [ 8220.133460] LustreError: 212004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8220.159336] LustreError: 212004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 8224.014014] LustreError: 211984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786097679 with bad export cookie 17749588747430981236 [ 8224.022733] LustreError: 211984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8224.031176] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8224.261309] Lustre: server umount lustre-MDT0001 complete [ 8228.791253] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 06:14:42 (1786097682) [ 8250.665771] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8261.862517] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8304.604909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8304.800767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8304.848294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8304.929254] Lustre: lustre-MDT0000: new disk, initializing [ 8305.099044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8310.482471] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8321.313418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8321.444600] Lustre: 218572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8321.699776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8326.140106] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8336.396642] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8342.960276] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8350.926304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8351.270236] Lustre: 220542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8353.119780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8358.550597] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8367.041762] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8371.531635] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8371.878428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8377.148301] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8377.349728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8383.974767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8383.979160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8383.986598] LustreError: Skipped 1 previous similar message [ 8383.987623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8383.987627] Lustre: Skipped 2 previous similar messages [ 8384.018023] Lustre: Skipped 3 previous similar messages [ 8385.476263] Lustre: server umount lustre-OST0000 complete [ 8395.720073] Lustre: server umount lustre-MDT0000 complete [ 8397.284317] LustreError: 218579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8397.313269] LustreError: 218579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8400.246366] LustreError: 218565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786097855 with bad export cookie 17749588747430982118 [ 8400.262085] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8401.068062] Lustre: server umount lustre-MDT0001 complete [ 8410.638555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8411.373718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8411.380683] Lustre: Skipped 4 previous similar messages [ 8417.502093] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8427.130735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8431.862926] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8439.307510] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8446.251206] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8455.195875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8462.844965] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8470.555978] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8482.784428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8482.791493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8482.806630] LustreError: Skipped 1 previous similar message [ 8482.814445] Lustre: Skipped 3 previous similar messages [ 8486.885847] Lustre: server umount lustre-OST0000 complete [ 8496.552243] Lustre: server umount lustre-MDT0000 complete [ 8500.101379] LustreError: 222740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786097955 with bad export cookie 17749588747430983000 [ 8500.115391] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8500.400278] Lustre: server umount lustre-MDT0001 complete [ 8609.789388] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8612.887379] Key type lgssc unregistered [ 8613.244248] LNet: 226419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8613.257991] LNetError: 226419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8613.273556] LNet: Removed LNI 192.168.201.106@tcp [ 8614.667173] Key type .llcrypt unregistered [ 8614.670722] Key type ._llcrypt unregistered [ 8647.204484] Key type ._llcrypt registered [ 8647.205962] Key type .llcrypt registered [ 8647.284195] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8663.472298] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8663.992626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8664.034509] alg: No test for adler32 (adler32-zlib) [ 8665.378206] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 8665.626607] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8667.271268] Key type lgssc registered [ 8668.612318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8704.126279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8704.156444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8705.436662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8705.458775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8705.530900] Lustre: lustre-MDT0000: new disk, initializing [ 8705.595987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8705.607404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8710.561631] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8722.934339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8723.066359] Lustre: 229779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8723.128153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8723.132665] Lustre: Skipped 1 previous similar message [ 8723.189262] Lustre: lustre-MDT0001: new disk, initializing [ 8723.243372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8723.279949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8723.299258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8728.515757] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8740.006963] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8747.139203] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8754.555312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8754.796508] Lustre: lustre-OST0000: new disk, initializing [ 8754.800126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8754.805040] Lustre: 231748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8754.884621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8761.985060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8762.013789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8762.208745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8762.241489] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8770.996357] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8775.863859] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8776.085553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8780.638726] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8780.913413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8782.822676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8782.839176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8782.839188] LustreError: Skipped 1 previous similar message [ 8782.874775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8787.945996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8787.958931] Lustre: Skipped 2 previous similar messages [ 8788.846583] Lustre: server umount lustre-OST0000 complete [ 8793.056930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8793.077242] Lustre: Skipped 1 previous similar message [ 8793.094934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8793.102478] Lustre: Skipped 1 previous similar message [ 8798.178187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8798.191295] Lustre: Skipped 1 previous similar message [ 8803.297864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8803.307587] Lustre: Skipped 1 previous similar message [ 8807.393563] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8807.785655] Lustre: server umount lustre-MDT0000 complete [ 8811.797055] LustreError: 231747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786098266 with bad export cookie 1406077810846488083 [ 8811.806383] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8812.256693] Lustre: server umount lustre-MDT0001 complete [ 8838.193518] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8850.356552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8850.609728] LustreError: 234705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8850.630822] LustreError: 234705:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8850.665733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8855.757238] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8856.037319] LustreError: 234706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8860.137318] LustreError: 234705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8864.201317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8864.469644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8870.188266] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8874.199947] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8882.651125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8882.906758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8891.391907] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8902.053782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8902.276483] Lustre: lustre-OST0001: new disk, initializing [ 8902.280621] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8902.287526] Lustre: 237015:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8902.420198] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8908.845658] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8909.402214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8909.411733] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8909.512066] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8919.575961] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8925.042265] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8939.809925] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 06:26:33 (1786098393) [ 8945.682670] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8945.689426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8945.701287] Lustre: Skipped 4 previous similar messages [ 8945.728538] Lustre: Skipped 3 previous similar messages [ 8950.138453] Lustre: server umount lustre-MDT0000 complete [ 8950.764233] LustreError: 234700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8950.788338] LustreError: 234700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8955.436787] LustreError: 234686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786098410 with bad export cookie 1406077810846488965 [ 8955.447430] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8955.450624] LustreError: 234686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8955.914560] LustreError: 234702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8955.958071] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8956.369845] Lustre: server umount lustre-MDT0001 complete [ 8961.996911] Lustre: server umount lustre-OST0000 complete [ 8967.523878] Lustre: server umount lustre-OST0001 complete [ 8974.902445] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8983.768559] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9018.310967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9018.579758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9018.601248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9018.670509] Lustre: lustre-MDT0000: new disk, initializing [ 9018.732210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9018.748902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9023.055806] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9033.938137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9034.068082] Lustre: 240957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9034.132465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9034.143203] Lustre: Skipped 1 previous similar message [ 9034.293613] Lustre: lustre-MDT0001: new disk, initializing [ 9034.521462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9034.541420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9039.299622] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9048.741349] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9055.046427] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9063.494224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9063.995228] Lustre: lustre-OST0000: new disk, initializing [ 9064.000533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9064.021614] Lustre: 242925:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9065.611129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9065.638480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9065.831690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9072.793959] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9085.743847] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9091.072959] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9091.385040] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9097.090149] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9097.495413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9101.794890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9101.798821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9101.802953] Lustre: Skipped 1 previous similar message [ 9101.806337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9101.810363] Lustre: Skipped 1 previous similar message [ 9106.822859] Lustre: server umount lustre-OST0000 complete [ 9112.043720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9112.077784] Lustre: Skipped 2 previous similar messages [ 9126.367704] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9126.596680] Lustre: server umount lustre-MDT0000 complete [ 9127.396732] LustreError: 240969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9127.422282] LustreError: 240969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9130.853304] LustreError: 240951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786098585 with bad export cookie 1406077810846491100 [ 9130.864310] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9130.880820] LustreError: 240951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9131.505118] Lustre: server umount lustre-MDT0001 complete [ 9143.878251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9144.743083] LustreError: 245147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9144.900782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9144.906303] Lustre: Skipped 2 previous similar messages [ 9149.587223] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9159.665333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9165.729375] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9178.442995] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9189.545340] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9199.494500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9206.892581] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9214.234204] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9363.437191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9363.439097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9363.446781] Lustre: Skipped 1 previous similar message [ 9363.447863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9363.447869] Lustre: Skipped 6 previous similar messages [ 9363.471433] LustreError: Skipped 1 previous similar message [ 9368.392289] Lustre: server umount lustre-OST0000 complete [ 9380.089820] Lustre: server umount lustre-MDT0000 complete [ 9383.905035] LustreError: 246107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9383.947078] LustreError: 246107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9384.474426] LustreError: 246994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786098839 with bad export cookie 1406077810846491982 [ 9384.480621] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9384.485804] LustreError: 246994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9384.807774] Lustre: server umount lustre-MDT0001 complete [ 9485.219359] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9488.288829] Key type lgssc unregistered [ 9488.709572] LNet: 248759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9488.715800] LNetError: 248759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9488.742130] LNet: Removed LNI 192.168.201.106@tcp [ 9490.356328] Key type .llcrypt unregistered [ 9490.360106] Key type ._llcrypt unregistered [ 9521.487683] Key type ._llcrypt registered [ 9521.491612] Key type .llcrypt registered [ 9521.655801] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9540.495859] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9542.090091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9542.137061] alg: No test for adler32 (adler32-zlib) [ 9543.312211] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 9543.834809] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 9545.647404] Key type lgssc registered [ 9547.880666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9611.066160] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9624.382621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9624.447032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9625.949576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9626.014420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9626.191485] Lustre: lustre-MDT0000: new disk, initializing [ 9626.282421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9626.324417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9632.543243] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9646.484674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9646.633576] Lustre: 252836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9646.695045] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9646.700424] Lustre: Skipped 1 previous similar message [ 9646.811469] Lustre: lustre-MDT0001: new disk, initializing [ 9647.064664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9647.143312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9647.174917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9652.516325] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9660.277086] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9670.252608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9670.759787] Lustre: lustre-OST0000: new disk, initializing [ 9670.793156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9670.818609] Lustre: 254466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9671.097384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9679.491091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9679.508340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9679.788697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9683.847079] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9698.068507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.310977] Lustre: lustre-OST0001: new disk, initializing [ 9698.314199] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9698.320151] Lustre: 255340:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9698.434330] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9705.674927] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9706.040845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9706.046567] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9706.186500] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9717.045319] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9726.033350] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9739.233181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9739.237339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9739.245584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9742.308891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9742.321582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9742.331783] Lustre: Skipped 1 previous similar message [ 9744.873140] Lustre: server umount lustre-MDT0000 complete [ 9750.177514] LustreError: 256109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786099205 with bad export cookie 10126224216277725488 [ 9750.192345] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9750.206617] LustreError: 256109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9750.704936] Lustre: server umount lustre-MDT0001 complete [ 9765.610944] Lustre: server umount lustre-OST0000 complete [ 9767.907821] Lustre: 250337:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786099207/real 1786099207] req@ffffa00a8255d180 x1872860543010304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786099223 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9767.940790] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9767.966942] Lustre: Skipped 1 previous similar message [ 9770.017450] Lustre: server umount lustre-OST0001 complete [ 9784.945974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9796.301892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9806.266569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9817.398518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9840.752260] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9851.699743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9851.715407] Lustre: lustre-MDT0000: reset Object Index mappings [ 9852.104989] LustreError: 260350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9852.111255] LustreError: 260350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9852.164251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9857.262652] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9857.516706] LustreError: 260351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9862.631873] LustreError: 260350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9867.759547] LustreError: 260351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9868.228990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9868.247725] Lustre: lustre-MDT0001: reset Object Index mappings [ 9868.576890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9875.394503] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9878.507248] Lustre: 261493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9886.475212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9886.955324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9893.109844] LustreError: 261848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9894.512705] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9897.224124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9902.588130] LustreError: 261847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9902.633731] LustreError: 261847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9906.406936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9911.896723] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9914.644602] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9926.356685] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9932.330754] Lustre: 263369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9940.449314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9940.457288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9940.471744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9940.477058] Lustre: Skipped 3 previous similar messages [ 9942.512512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9942.521363] Lustre: Skipped 2 previous similar messages [ 9947.637298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9947.646472] Lustre: Skipped 1 previous similar message [ 9954.272305] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9956.591919] Lustre: server umount lustre-MDT0000 complete [ 9957.857829] LustreError: 260350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9957.898155] LustreError: 260350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9960.498952] LustreError: 260333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786099415 with bad export cookie 10126224216277732278 [ 9960.502865] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9960.505819] LustreError: 260333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9960.993048] Lustre: server umount lustre-MDT0001 complete [ 9976.121628] Lustre: server umount lustre-OST0000 complete [ 9978.912813] Lustre: 250337:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786099418/real 1786099418] req@ffffa00a8dc86d80 x1872860543070080/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786099434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9978.941915] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9978.952339] Lustre: Skipped 3 previous similar messages [ 9982.047558] Lustre: 250337:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786099421/real 1786099421] req@ffffa00a93b3d880 x1872860543070336/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786099437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9983.841759] Lustre: server umount lustre-OST0001 complete [10005.554298] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 06:44:19 (1786099459) [10023.582332] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [10038.764714] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10079.430616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10079.726957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10079.771355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10079.876635] Lustre: lustre-MDT0000: new disk, initializing [10080.001706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10080.010376] Lustre: Skipped 1 previous similar message [10080.024529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10085.232370] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10095.653362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10095.768440] Lustre: 267648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10095.818526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10095.823672] Lustre: Skipped 1 previous similar message [10095.883342] Lustre: lustre-MDT0001: new disk, initializing [10096.026665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10096.047506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10100.649090] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10111.157441] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10118.042779] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10125.695332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10126.056512] Lustre: lustre-OST0000: new disk, initializing [10126.063602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10126.069289] Lustre: 269616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10128.025168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10128.032546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10128.200269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10133.075389] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10146.192690] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10151.407509] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10151.838310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10158.444649] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10158.592860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10164.193256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10164.197763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10164.227346] LustreError: Skipped 1 previous similar message [10164.233685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10164.233696] Lustre: Skipped 7 previous similar messages [10164.347281] Lustre: Skipped 1 previous similar message [10168.085951] Lustre: server umount lustre-OST0000 complete [10178.023859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10184.675696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10184.679995] Lustre: Skipped 5 previous similar messages [10189.279392] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10189.598959] Lustre: server umount lustre-MDT0000 complete [10189.798018] LustreError: 267657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10189.806234] LustreError: 267657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10193.800781] LustreError: 267643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786099648 with bad export cookie 10126224216277734371 [10193.801866] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10193.814335] LustreError: 267643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10194.350353] Lustre: server umount lustre-MDT0001 complete [10220.619353] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10233.276484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10234.024769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10234.028727] Lustre: Skipped 2 previous similar messages [10238.800863] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10248.595077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10254.809795] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10259.539132] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10269.134199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10279.683371] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10292.785815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10293.218816] Lustre: lustre-OST0001: new disk, initializing [10293.222498] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10293.228814] Lustre: 274888:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10294.383955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10294.386862] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10294.517593] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10301.658268] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10311.028625] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10314.749898] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10320.439058] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10320.665357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10330.616786] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10330.643377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10330.658304] Lustre: Skipped 5 previous similar messages [10330.685528] Lustre: Skipped 3 previous similar messages [10334.635295] Lustre: server umount lustre-MDT0000 complete [10335.716760] LustreError: 275963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10335.759703] LustreError: 275963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10339.853825] LustreError: 273712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786099794 with bad export cookie 10126224216277735253 [10339.874782] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10339.878495] LustreError: 273712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10340.492362] Lustre: server umount lustre-MDT0001 complete [10355.394251] Lustre: server umount lustre-OST0000 complete [10356.191111] Lustre: 250336:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786099795/real 1786099795] req@ffffa00a88a3a680 x1872860543189888/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786099811 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10359.243658] Lustre: server umount lustre-OST0001 complete [10377.581936] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10391.178854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10391.888812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10391.896453] Lustre: Skipped 3 previous similar messages [10398.692563] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10412.623846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10419.083995] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10422.195605] Lustre: 279087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10430.222222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10436.024232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10439.451784] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10449.599816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10455.051223] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10456.708218] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10464.285492] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10468.118318] Lustre: 280961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10474.811402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10474.941978] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10486.360857] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 06:52:20 (1786099940) [10505.184665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10505.197992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10505.221737] Lustre: Skipped 1 previous similar message [10505.232035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10508.967943] Lustre: server umount lustre-MDT0000 complete [10511.346856] LustreError: 277946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10511.379129] LustreError: 277946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [10513.681504] LustreError: 277927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786099968 with bad export cookie 10126224216277742050 [10513.689947] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10513.696626] LustreError: 277927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10520.654524] Lustre: server umount lustre-MDT0001 complete [10532.781917] Lustre: server umount lustre-OST0000 complete [10538.674361] Lustre: server umount lustre-OST0001 complete [10548.336240] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [10561.416389] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10614.529318] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10629.345966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10629.520543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10629.542518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10629.613541] Lustre: lustre-MDT0000: new disk, initializing [10629.667951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10634.615399] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10645.316243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10645.426332] Lustre: 285210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10645.862795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10650.581827] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10661.809697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10662.072367] Lustre: 286840:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10662.167322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10662.171853] Lustre: Skipped 5 previous similar messages [10663.901870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10668.780234] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10680.825321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10681.040909] Lustre: 287710:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10682.519670] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10682.531788] Lustre: Skipped 1 previous similar message [10682.681373] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10689.079763] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10700.851364] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10705.131940] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10705.140202] Lustre: Skipped 1 previous similar message [10722.977435] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 06:56:16 (1786100176) [10727.908394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10727.924637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10727.933849] Lustre: Skipped 5 previous similar messages [10728.133616] Lustre: server umount lustre-MDT0000 complete [10732.349111] LustreError: 286839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786100187 with bad export cookie 10126224216277749470 [10732.361136] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10732.372503] LustreError: 286839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10732.830726] Lustre: server umount lustre-MDT0001 complete [10748.279711] Lustre: server umount lustre-OST0000 complete [10750.443249] Lustre: 250336:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786100189/real 1786100189] req@ffffa00a94313b80 x1872860543351808/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786100205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10752.660385] Lustre: server umount lustre-OST0001 complete [10761.875793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10767.238409] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10767.851311] LustreError: 289958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10767.874704] LustreError: 289958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10777.260327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10782.508499] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10790.798275] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10796.130737] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10802.810171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10809.078747] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10817.354620] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10843.920297] loop1: detected capacity change from 0 to 204800000 [10844.130059] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10844.147350] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10844.161426] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10849.716346] loop1: detected capacity change from 0 to 204800000 [10855.201590] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10855.281288] LustreError: 289949:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs18024: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10855.304828] LustreError: 289949:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs18024 [10855.327628] LustreError: 289949:0:(mgs_handler.c:517:mgs_target_reg()) fs18024-MDT0001: Error checking target index: rc = -13 [10855.341550] LustreError: fs18024-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10855.347757] LustreError: 293695:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10855.354518] LustreError: 293695:0:(tgt_mount.c:2129:server_put_super()) no obd fs18024-MDT0001 [10855.359144] LustreError: 293695:0:(tgt_mount.c:116:server_deregister_mount()) fs18024-MDT0001 not registered [10855.386898] Lustre: server umount fs18024-MDT0001 complete [10855.389657] LustreError: 293695:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10864.508896] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10864.599349] Lustre: Setting parameter fs18024-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs18024-MDT0001 [10864.654833] Lustre: srv-fs18024-MDT0001: No data found on store. Initialize space. [10864.657926] Lustre: Skipped 4 previous similar messages [10864.725434] Lustre: fs18024-MDT0001: new disk, initializing [10864.728750] Lustre: Skipped 3 previous similar messages [10864.853798] Lustre: srv-fs18024-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10864.862778] Lustre: 294162:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18024-MDT0001-mdtlov: can't create new llog: rc = -115 [10864.880141] LustreError: 294162:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18024-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10869.758498] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10876.718996] Lustre: server umount fs18024-MDT0001 complete [10885.097850] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10889.909357] loop1: detected capacity change from 0 to 204800000 [10895.708763] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10896.110690] Lustre: 295731:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18024-MDT0001-mdtlov: can't create new llog: rc = -115 [10896.132522] LustreError: 295731:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18024-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10900.979708] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10906.380774] Lustre: server umount fs18024-MDT0001 complete [10920.833128] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [10920.861199] Lustre: Skipped 8 previous similar messages [10920.934481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10920.947806] LustreError: Skipped 1 previous similar message [10920.951066] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10920.979675] Lustre: Skipped 5 previous similar messages [10933.215327] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10933.484364] Lustre: server umount lustre-MDT0000 complete [10941.901031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10942.114671] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10947.068518] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10947.569883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10955.512867] loop1: detected capacity change from 0 to 204800000 [10961.688199] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10961.920610] Lustre: 298178:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs18024-MDT0001-mdtlov: can't create new llog: rc = -115 [10961.926445] LustreError: 298178:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs18024-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10968.138807] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10973.636838] Lustre: server umount fs18024-MDT0001 complete [10997.735763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11001.476097] Lustre: server umount lustre-OST0000 complete [11011.496319] Lustre: server umount lustre-MDT0000 complete [11015.332984] LustreError: 289941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786100470 with bad export cookie 10126224216277752151 [11015.340069] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11015.353038] LustreError: 289941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11016.052840] Lustre: server umount lustre-MDT0001 complete [11027.577474] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [11030.606280] Key type lgssc unregistered [11030.921929] LNet: 300533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11030.929397] LNetError: 300533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11030.979135] LNet: Removed LNI 192.168.201.106@tcp [11032.144230] Key type .llcrypt unregistered [11032.146157] Key type ._llcrypt unregistered [11058.003095] Key type ._llcrypt registered [11058.005454] Key type .llcrypt registered [11058.128902] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11059.670269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11059.922352] alg: No test for adler32 (adler32-zlib) [11061.018694] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [11061.341829] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [11062.991223] Key type lgssc registered [11064.031557] Lustre: Echo OBD driver; http://www.lustre.org/ [11072.794882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11072.819220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11074.530760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11079.339881] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11084.768604] LustreError: 301808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11084.790325] LustreError: 301808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11088.616212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11089.200388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11093.949627] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11102.307653] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11108.138255] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11115.637138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11116.297707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11119.345914] LustreError: 303640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11124.726306] LustreError: 303639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11124.797860] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11129.845342] LustreError: 303640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11133.342529] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11133.986070] LustreError: 303641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11134.023902] LustreError: 303641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11144.163323] LustreError: 303640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11144.190620] LustreError: 303640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11155.427564] loop1: detected capacity change from 0 to 204800000 [11155.761932] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11155.786315] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11155.798094] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11160.916628] LustreError: 303641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11160.952340] LustreError: 303641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [11163.037373] loop1: detected capacity change from 0 to 204800000 [11169.315514] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11169.466351] Lustre: fs16853-OST0000: new disk, initializing [11169.471310] Lustre: srv-fs16853-OST0000: No data found on store. Initialize space. [11169.480380] Lustre: 305546:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11169.564476] Lustre: fs16853-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11175.188763] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11183.988802] Lustre: server umount fs16853-OST0000 complete [11195.366426] LustreError: 303641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11195.384854] LustreError: 303641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [11196.862550] loop1: detected capacity change from 0 to 204800000 [11200.497557] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11201.999548] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11202.248162] Lustre: fs16853-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11208.265127] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11215.847381] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11217.070656] loop2: detected capacity change from 0 to 204800000 [11217.346186] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11217.364753] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11217.383912] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11223.165544] loop2: detected capacity change from 0 to 204800000 [11229.202449] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11229.300630] LustreError: fs16853-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11229.313110] LustreError: fs16853-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11229.320462] LustreError: 308585:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11229.325760] LustreError: 308585:0:(tgt_mount.c:2129:server_put_super()) no obd fs16853-OST0001 [11229.335465] LustreError: 308585:0:(tgt_mount.c:116:server_deregister_mount()) fs16853-OST0001 not registered [11229.413664] Lustre: server umount fs16853-OST0001 complete [11229.416905] LustreError: 308585:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11241.571257] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11241.723212] Lustre: fs16853-OST0001: new disk, initializing [11241.727991] Lustre: srv-fs16853-OST0001: No data found on store. Initialize space. [11241.738675] Lustre: 309137:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11241.876856] Lustre: fs16853-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11248.390946] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11257.054966] Lustre: server umount fs16853-OST0000 complete [11261.936689] LustreError: 303640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11261.973237] LustreError: 303640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11266.761944] Lustre: server umount fs16853-OST0001 complete [11279.559297] loop1: detected capacity change from 0 to 204800000 [11286.156733] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11286.525432] Lustre: fs16853-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11293.247915] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11301.904070] loop2: detected capacity change from 0 to 204800000 [11310.162438] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11318.912839] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11330.155470] Lustre: server umount fs16853-OST0000 complete [11339.774234] Lustre: server umount fs16853-OST0001 complete [11354.082812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11354.085472] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11354.096224] Lustre: Skipped 1 previous similar message [11354.112413] LustreError: Skipped 1 previous similar message [11354.118515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11354.125918] Lustre: Skipped 1 previous similar message [11357.841441] Lustre: server umount lustre-OST0000 complete [11364.323805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11364.336293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11364.343638] Lustre: Skipped 1 previous similar message [11368.053776] Lustre: server umount lustre-MDT0000 complete [11371.851453] LustreError: 301788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786100826 with bad export cookie 4129235186524559599 [11371.854444] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11371.862745] LustreError: 301788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11372.150377] Lustre: server umount lustre-MDT0001 complete [11383.893176] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [11386.445360] Key type lgssc unregistered [11386.784506] LNet: 315061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11386.794430] LNetError: 315061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11387.822312] LNet: Removed LNI 192.168.201.106@tcp [11388.679178] Key type .llcrypt unregistered [11388.682476] Key type ._llcrypt unregistered [11421.355830] Key type ._llcrypt registered [11421.358353] Key type .llcrypt registered [11421.465860] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [11437.139348] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11438.501961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11438.819973] alg: No test for adler32 (adler32-zlib) [11439.852399] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [11440.283697] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [11441.920160] Key type lgssc registered [11443.251110] Lustre: Echo OBD driver; http://www.lustre.org/ [11482.324826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11482.338156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11483.650349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11483.693625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11483.822387] Lustre: lustre-MDT0000: new disk, initializing [11483.883845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11483.914764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11488.363649] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11500.215748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11500.420463] Lustre: 318923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11500.540911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11500.548494] Lustre: Skipped 1 previous similar message [11500.673571] Lustre: lustre-MDT0001: new disk, initializing [11500.803602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11500.872176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11500.888617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11505.868299] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11515.692592] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11521.703277] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11529.196121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11529.544625] Lustre: lustre-OST0000: new disk, initializing [11529.548808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11529.552744] Lustre: 320891:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11529.691266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11537.269520] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11538.511694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11538.527440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11538.631918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11547.161524] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11549.297854] Lustre: Mounted lustre-client [11550.383180] Lustre: Unmounted lustre-client [11564.527065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11564.533479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11564.554448] Lustre: Skipped 1 previous similar message [11564.571405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11566.926822] Lustre: server umount lustre-OST0000 complete [11572.704954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11572.710318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11572.726952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11572.744198] Lustre: Skipped 1 previous similar message [11574.755786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11574.760247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11577.017504] Lustre: server umount lustre-MDT0000 complete [11580.828527] LustreError: 318915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101035 with bad export cookie 11463279587566266949 [11580.834171] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11580.836858] LustreError: 318915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11581.183799] Lustre: server umount lustre-MDT0001 complete [11591.959527] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [11594.431278] Key type lgssc unregistered [11609.248426] LNet: 1 local NIs in recovery (showing 1): 192.168.201.106@tcp [11610.015504] LNet: 323052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11610.022245] LNetError: 323052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11679.663756] LNet: Removed LNI 192.168.201.106@tcp [11680.558744] Key type .llcrypt unregistered [11680.561062] Key type ._llcrypt unregistered [11698.306705] Key type ._llcrypt registered [11698.310917] Key type .llcrypt registered [11698.405032] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [11713.143282] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11714.440594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11714.474439] alg: No test for adler32 (adler32-zlib) [11715.561588] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [11715.797850] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [11717.455170] Key type lgssc registered [11718.557891] Lustre: Echo OBD driver; http://www.lustre.org/ [11753.808939] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:13:27 (1786101207) [11761.117993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11761.171224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11762.444212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11762.519195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11762.644644] Lustre: lustre-MDT0000: new disk, initializing [11762.718540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11762.736877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11767.544532] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11778.486282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11778.607832] Lustre: 326275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11778.673711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11778.678262] Lustre: Skipped 1 previous similar message [11778.773628] Lustre: lustre-MDT0001: new disk, initializing [11778.889569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11778.925965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11778.947645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11783.622163] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11794.642715] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11801.221430] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11808.122699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11808.562261] Lustre: lustre-OST0000: new disk, initializing [11808.566510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11808.577245] Lustre: 328243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11808.714912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11814.917465] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11816.069097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11816.081691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11816.182369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11824.133425] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11842.020219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11842.036136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11842.059167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11842.069407] Lustre: Skipped 1 previous similar message [11843.578176] Lustre: server umount lustre-OST0000 complete [11847.142524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11847.158081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11847.167572] Lustre: Skipped 2 previous similar messages [11847.181907] Lustre: Skipped 1 previous similar message [11852.257967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11852.264739] Lustre: Skipped 1 previous similar message [11853.097519] Lustre: server umount lustre-MDT0000 complete [11857.019450] LustreError: 326269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101312 with bad export cookie 13146075932970519827 [11857.025228] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11857.028275] LustreError: 326269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11857.357613] Lustre: server umount lustre-MDT0001 complete [11868.246632] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [11870.867749] Key type lgssc unregistered [11871.167463] LNet: 330153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11871.175715] LNetError: 330153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11871.189885] LNet: Removed LNI 192.168.201.106@tcp [11872.099791] Key type .llcrypt unregistered [11872.106366] Key type ._llcrypt unregistered [11894.802457] Key type ._llcrypt registered [11894.805351] Key type .llcrypt registered [11894.978676] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11896.654117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11896.766278] alg: No test for adler32 (adler32-zlib) [11898.278332] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [11898.458603] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [11900.146059] Key type lgssc registered [11901.933471] Lustre: Echo OBD driver; http://www.lustre.org/ [11911.467178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11911.499867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11912.838579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11917.919937] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11922.407678] LustreError: 331430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11922.425440] LustreError: 331430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11927.020887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11927.463106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11931.911908] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11939.487919] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11944.518827] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11951.375789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11951.750630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11958.208863] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11965.561769] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11975.205549] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11976.751340] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11983.706773] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [11983.842112] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11983.846818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11983.861223] Lustre: Skipped 1 previous similar message [11986.583304] Lustre: server umount lustre-MDT0000 complete [11988.820641] LustreError: 331425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11990.430614] LustreError: 331409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101445 with bad export cookie 15095165709954247368 [11990.448953] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11993.930581] LustreError: 331424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11993.961542] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [11993.965713] LustreError: 331424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11993.985981] Lustre: Skipped 3 previous similar messages [11994.089914] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11995.123055] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11995.129521] Lustre: Skipped 1 previous similar message [11996.910211] Lustre: server umount lustre-MDT0001 complete [12000.757448] Lustre: server umount lustre-OST0000 complete [12010.297176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12015.244065] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12018.994108] Lustre: 335611:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.106@tcp [12021.702299] Lustre: server umount MGS complete [12031.137340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12031.492272] LustreError: 336182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12031.513701] LustreError: 336182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12031.582353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12036.265737] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12036.582724] LustreError: 336183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.623138] LustreError: 336183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12046.817285] LustreError: 336182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12046.836256] LustreError: 336182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12047.336604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12047.735561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12052.952548] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12062.345580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12062.800387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12069.761070] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12076.925500] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12078.525368] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12084.507670] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12085.899742] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12091.372380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12091.417300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12093.035507] Lustre: server umount lustre-OST0000 complete [12099.041158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12099.046077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12099.064034] Lustre: Skipped 1 previous similar message [12099.071410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12099.080728] Lustre: Skipped 1 previous similar message [12111.328214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12111.663882] Lustre: server umount lustre-MDT0000 complete [12111.846474] LustreError: 336182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12111.854608] LustreError: 336182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12115.385325] LustreError: 336163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101570 with bad export cookie 15095165709954248208 [12115.389568] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12115.400658] LustreError: 336163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12115.905972] Lustre: server umount lustre-MDT0001 complete [12126.378714] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12129.166888] Key type lgssc unregistered [12129.390733] LNet: 339623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12129.398326] LNetError: 339623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12129.412265] LNet: Removed LNI 192.168.201.106@tcp [12130.220734] Key type .llcrypt unregistered [12130.225771] Key type ._llcrypt unregistered [12150.808644] Key type ._llcrypt registered [12150.811461] Key type .llcrypt registered [12150.915221] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [12167.157661] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12168.567700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12168.790615] alg: No test for adler32 (adler32-zlib) [12169.849884] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [12170.104478] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [12171.768821] Key type lgssc registered [12173.205490] Lustre: Echo OBD driver; http://www.lustre.org/ [12212.272671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12212.315304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12213.534976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12213.559242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12213.621846] Lustre: lustre-MDT0000: new disk, initializing [12213.693269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12213.715975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12217.959467] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12228.278286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12228.392419] Lustre: 342801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12228.440683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12228.444454] Lustre: Skipped 1 previous similar message [12228.517858] Lustre: lustre-MDT0001: new disk, initializing [12228.620467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12228.673819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12228.686231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12233.525478] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12242.646712] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12247.993774] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12254.352133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12254.545539] Lustre: lustre-OST0000: new disk, initializing [12254.552587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12254.567755] Lustre: 344771:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12254.684275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12260.767849] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12264.020052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12264.034499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12264.206181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12270.184785] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12274.916505] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12275.106178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12279.289517] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12279.452420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12284.903829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12284.908333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12284.911015] LustreError: Skipped 1 previous similar message [12284.912176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12284.941144] Lustre: Skipped 1 previous similar message [12287.216076] Lustre: server umount lustre-OST0000 complete [12295.138373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12295.143161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12295.151288] Lustre: Skipped 1 previous similar message [12295.165716] Lustre: Skipped 1 previous similar message [12296.546533] Lustre: server umount lustre-MDT0000 complete [12299.860496] LustreError: 343597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101754 with bad export cookie 13017317097828331014 [12299.873833] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12300.342576] Lustre: server umount lustre-MDT0001 complete [12308.142863] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 07:22:42 (1786101762) [12326.744436] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [12338.559391] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12375.046928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12375.311664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12375.346487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12375.452730] Lustre: lustre-MDT0000: new disk, initializing [12375.576811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12375.598470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12381.001453] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12391.177967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12391.261325] Lustre: 349528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12391.368437] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12391.370815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12391.376770] Lustre: Skipped 1 previous similar message [12391.441532] Lustre: lustre-MDT0001: new disk, initializing [12391.519663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12391.550146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12391.556144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12395.907774] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12405.151421] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12410.468794] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12417.008733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12417.377248] Lustre: lustre-OST0000: new disk, initializing [12417.383398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12417.393970] Lustre: 351499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12417.529645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12423.853994] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12425.812742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12425.821175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12425.897954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12434.068684] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12477.422159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12477.422228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12477.430061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12477.432540] LustreError: Skipped 1 previous similar message [12477.454996] Lustre: Skipped 1 previous similar message [12482.529936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12491.231533] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12491.557882] Lustre: server umount lustre-OST0000 complete [12497.894742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12497.905147] Lustre: Skipped 1 previous similar message [12497.910171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12497.915767] Lustre: Skipped 4 previous similar messages [12502.355790] Lustre: server umount lustre-MDT0000 complete [12506.859089] LustreError: 349521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786101961 with bad export cookie 13017317097828331896 [12506.866709] LustreError: 349521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12506.872902] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12507.375702] Lustre: server umount lustre-MDT0001 complete [12521.320287] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12525.171176] Key type lgssc unregistered [12525.538040] LNet: 353674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12525.544172] LNetError: 353674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12525.582483] LNet: Removed LNI 192.168.201.106@tcp [12526.449645] Key type .llcrypt unregistered [12526.452741] Key type ._llcrypt unregistered [12624.056694] Key type ._llcrypt registered [12624.058296] Key type .llcrypt registered [12624.147217] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12624.353930] Key type .llcrypt unregistered [12624.355308] Key type ._llcrypt unregistered [12706.531864] Key type ._llcrypt registered [12706.535264] Key type .llcrypt registered [12706.653699] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12707.007348] Key type .llcrypt unregistered [12707.009192] Key type ._llcrypt unregistered [12743.425437] Key type ._llcrypt registered [12743.438222] Key type .llcrypt registered [12743.660170] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12744.133362] Key type .llcrypt unregistered [12744.134451] Key type ._llcrypt unregistered [12784.119651] Key type ._llcrypt registered [12784.122231] Key type .llcrypt registered [12784.188828] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12784.522376] Key type .llcrypt unregistered [12784.524202] Key type ._llcrypt unregistered [12861.510731] Key type ._llcrypt registered [12861.514236] Key type .llcrypt registered [12861.686962] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12862.075377] Key type .llcrypt unregistered [12862.078578] Key type ._llcrypt unregistered [12901.841610] Key type ._llcrypt registered [12901.843099] Key type .llcrypt registered [12901.940349] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12902.225647] Key type .llcrypt unregistered [12902.231257] Key type ._llcrypt unregistered [12939.350804] Key type ._llcrypt registered [12939.353568] Key type .llcrypt registered [12939.597401] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12940.137779] Key type .llcrypt unregistered [12940.146587] Key type ._llcrypt unregistered [12977.319390] Key type ._llcrypt registered [12977.325413] Key type .llcrypt registered [12977.455956] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12977.740544] Key type .llcrypt unregistered [12977.742616] Key type ._llcrypt unregistered [13004.729530] Key type ._llcrypt registered [13004.732772] Key type .llcrypt registered [13004.862398] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13005.096493] Key type .llcrypt unregistered [13005.097979] Key type ._llcrypt unregistered [13037.921887] Key type ._llcrypt registered [13037.931024] Key type .llcrypt registered [13038.072849] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13038.447537] Key type .llcrypt unregistered [13038.450960] Key type ._llcrypt unregistered [13103.188987] Key type ._llcrypt registered [13103.193531] Key type .llcrypt registered [13103.283828] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13103.544423] Key type .llcrypt unregistered [13103.549751] Key type ._llcrypt unregistered [13136.499412] Key type ._llcrypt registered [13136.503715] Key type .llcrypt registered [13136.610160] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13136.911394] Key type .llcrypt unregistered [13136.912884] Key type ._llcrypt unregistered [13170.152659] Key type ._llcrypt registered [13170.161743] Key type .llcrypt registered [13170.360656] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13170.716517] Key type .llcrypt unregistered [13170.719399] Key type ._llcrypt unregistered [13235.041663] Key type ._llcrypt registered [13235.044225] Key type .llcrypt registered [13235.184422] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [13254.518654] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13255.986492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13256.051459] alg: No test for adler32 (adler32-zlib) [13257.139963] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [13257.362669] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [13258.994098] Key type lgssc registered [13260.312334] Lustre: Echo OBD driver; http://www.lustre.org/ [13293.208942] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13293.231023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13294.585331] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13294.658814] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13294.744567] Lustre: tst250fs-MDT0000: new disk, initializing [13294.840070] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13294.855643] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13299.597965] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13311.224643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13311.362979] Lustre: 373994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13311.485210] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13311.497178] Lustre: Skipped 1 previous similar message [13311.617951] Lustre: tst250fs-MDT0001: new disk, initializing [13311.721559] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13311.760291] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13311.770339] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13316.229595] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13321.442405] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13327.981343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13328.321684] Lustre: tst250fs-OST0000: new disk, initializing [13328.326348] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13328.335319] Lustre: 375629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13328.477919] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13333.118186] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13333.124742] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13333.264288] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13335.286874] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13347.773312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13348.084645] Lustre: tst250fs-OST0001: new disk, initializing [13348.089807] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13348.102882] Lustre: 376502:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13348.242684] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13354.581922] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13358.301939] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13358.334690] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13358.504169] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13361.293605] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13364.590604] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13364.598817] Lustre: Skipped 2 previous similar messages [13383.651959] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13383.659155] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13383.676613] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13384.164617] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13384.185141] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13384.205321] Lustre: Skipped 2 previous similar messages [13384.209765] Lustre: Skipped 2 previous similar messages [13389.287258] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13389.302314] Lustre: Skipped 3 previous similar messages [13389.458391] Lustre: server umount tst250fs-MDT0000 complete [13392.959043] LustreError: 373989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786102848 with bad export cookie 811946247705593273 [13392.964043] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13392.971968] LustreError: 373989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13393.253117] Lustre: server umount tst250fs-MDT0001 complete [13406.118244] Lustre: server umount tst250fs-OST0000 complete [13419.102675] Lustre: server umount tst250fs-OST0001 complete [13438.467761] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [13446.723913] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13479.126887] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 07:42:13 (1786102933) [13481.022204] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13502.428754] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [13511.825095] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13543.240336] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13244 sec ========= 07:43:16 (1786102996) [13545.560066] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:43:18 (1786102998) === [13549.195335] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:43:23 (1786103003) === [13586.206895] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13588.863950] Key type lgssc unregistered [13589.320660] LNet: 385633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13589.337587] LNetError: 385633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13589.358753] LNet: Removed LNI 192.168.201.106@tcp [13590.291140] Key type .llcrypt unregistered [13590.292414] Key type ._llcrypt unregistered