[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480704927 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003166] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010142] LSM: Security Framework initializing [ 0.012006] Yama: becoming mindful. [ 0.013048] SELinux: Initializing. [ 0.014084] *** VALIDATE selinux *** [ 0.022542] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027128] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030121] *** VALIDATE tmpfs *** [ 0.032200] *** VALIDATE proc *** [ 0.033250] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.041944] debug: unmapping init [mem 0xffffffff9f459000-0xffffffff9f460fff] [ 0.043915] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044747] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046017] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048015] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052296] rcu: Hierarchical SRCU implementation. [ 0.054435] smp: Bringing up secondary CPUs ... [ 0.055613] x86: Booting SMP configuration: [ 0.056026] .... node #0, CPUs: #1 #2 #3 [ 0.061182] smp: Brought up 1 node, 4 CPUs [ 0.063018] smpboot: Max logical packages: 1 [ 0.064014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.121646] node 0 deferred pages initialised in 55ms [ 0.124248] devtmpfs: initialized [ 0.125200] x86/mm: Memory block size: 128MB [ 0.127417] gcov: version magic: 0x41383552 [ 0.130092] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.132171] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.134357] pinctrl core: initialized pinctrl subsystem [ 0.136138] [ 0.136849] ************************************************************* [ 0.138012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140011] ** ** [ 0.142010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.144015] ** ** [ 0.146010] ** This means that this kernel is built to expose internal ** [ 0.148012] ** IOMMU data structures, which may compromise security on ** [ 0.150011] ** your system. ** [ 0.152014] ** ** [ 0.155018] ** If you see this message and you are not debugging the ** [ 0.157012] ** kernel, report this immediately to your vendor! ** [ 0.159013] ** ** [ 0.161010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163013] ************************************************************* [ 0.165780] NET: Registered protocol family 16 [ 0.168494] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.178037] cpuidle: using governor menu [ 0.179705] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182454] PCI: Using configuration type 1 for base access [ 0.184096] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193060] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199056] cryptd: max_cpu_qlen set to 1000 [ 0.202305] ACPI: Added _OSI(Module Device) [ 0.203015] ACPI: Added _OSI(Processor Device) [ 0.205016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212563] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.219442] ACPI: Interpreter enabled [ 0.220047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221012] ACPI: Using IOAPIC for interrupt routing [ 0.222103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.224360] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232844] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.234037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.236022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.239088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.243259] acpiphp: Slot [2] registered [ 0.244125] acpiphp: Slot [5] registered [ 0.245117] acpiphp: Slot [6] registered [ 0.247156] acpiphp: Slot [7] registered [ 0.248085] acpiphp: Slot [8] registered [ 0.249097] acpiphp: Slot [9] registered [ 0.250083] acpiphp: Slot [10] registered [ 0.251102] acpiphp: Slot [3] registered [ 0.252087] acpiphp: Slot [4] registered [ 0.253092] acpiphp: Slot [11] registered [ 0.255085] acpiphp: Slot [12] registered [ 0.256088] acpiphp: Slot [13] registered [ 0.257069] acpiphp: Slot [14] registered [ 0.258064] acpiphp: Slot [15] registered [ 0.259080] acpiphp: Slot [16] registered [ 0.260062] acpiphp: Slot [17] registered [ 0.261079] acpiphp: Slot [18] registered [ 0.262077] acpiphp: Slot [19] registered [ 0.263072] acpiphp: Slot [20] registered [ 0.264068] acpiphp: Slot [21] registered [ 0.265103] acpiphp: Slot [22] registered [ 0.266079] acpiphp: Slot [23] registered [ 0.267093] acpiphp: Slot [24] registered [ 0.268066] acpiphp: Slot [25] registered [ 0.269093] acpiphp: Slot [26] registered [ 0.271176] acpiphp: Slot [27] registered [ 0.273130] acpiphp: Slot [28] registered [ 0.274107] acpiphp: Slot [29] registered [ 0.276106] acpiphp: Slot [30] registered [ 0.277120] acpiphp: Slot [31] registered [ 0.279073] PCI host bridge to bus 0000:00 [ 0.281022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.283023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.285028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.288022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.290017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.291026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.293178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.295834] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.300219] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.309809] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.315569] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.318022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.321021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.324030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.328041] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.331232] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.334060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.337979] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.343025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.358022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.365036] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.369148] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.376016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.382014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.396015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.405020] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.414029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.425032] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.463023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.474386] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.483019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.490021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.505021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.515662] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.524032] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.534028] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.553024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.565172] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.572019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.578015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.598018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.617229] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.627015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.633014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.649016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.659537] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.665459] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.668418] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.670506] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.672263] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.677093] iommu: Default domain type: Passthrough [ 0.679479] SCSI subsystem initialized [ 0.681152] ACPI: bus type USB registered [ 0.682113] usbcore: registered new interface driver usbfs [ 0.685207] usbcore: registered new interface driver hub [ 0.687100] usbcore: registered new device driver usb [ 0.689197] pps_core: LinuxPPS API ver. 1 registered [ 0.691012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.694286] PTP clock support registered [ 0.696119] EDAC MC: Ver: 3.0.0 [ 0.698133] PCI: Using ACPI for IRQ routing [ 0.699514] NetLabel: Initializing [ 0.701014] NetLabel: domain hash size = 128 [ 0.703013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.705081] NetLabel: unlabeled traffic allowed by default [ 0.707088] vgaarb: loaded [ 0.709313] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.711015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.716464] clocksource: Switched to clocksource kvm-clock [ 0.818697] VFS: Disk quotas dquot_6.6.0 [ 0.819777] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.822326] *** VALIDATE ramfs *** [ 0.823165] *** VALIDATE hugetlbfs *** [ 0.824197] pnp: PnP ACPI init [ 0.826188] pnp: PnP ACPI: found 6 devices [ 0.844890] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.846904] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.847990] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.849238] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.850384] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.851429] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.852894] NET: Registered protocol family 2 [ 0.854502] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.857982] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.859846] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.863197] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.864984] TCP: Hash tables configured (established 65536 bind 65536) [ 0.867458] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.869412] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.871385] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.873800] NET: Registered protocol family 1 [ 0.875703] RPC: Registered named UNIX socket transport module. [ 0.877851] RPC: Registered udp transport module. [ 0.879144] RPC: Registered tcp transport module. [ 0.880224] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.881732] NET: Registered protocol family 44 [ 0.883219] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.884689] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.887930] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.890027] PCI: CLS 0 bytes, default 64 [ 0.892123] Unpacking initramfs... [ 2.265608] debug: unmapping init [mem 0xffff9a6e7cc54000-0xffff9a6e7ffbffff] [ 2.268827] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.270776] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.273522] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.737283] Initialise system trusted keyrings [ 2.739680] Key type blacklist registered [ 2.741458] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.750417] zbud: loaded [ 2.753451] *** VALIDATE nfs *** [ 2.754382] *** VALIDATE nfs4 *** [ 2.755993] pstore: using deflate compression [ 2.759809] Platform Keyring initialized [ 2.896867] NET: Registered protocol family 38 [ 2.898927] Key type asymmetric registered [ 2.900548] Asymmetric key parser 'x509' registered [ 2.902629] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.905767] io scheduler mq-deadline registered [ 2.907658] io scheduler kyber registered [ 2.909453] io scheduler bfq registered [ 2.913224] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.916670] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.919913] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.922937] ACPI: Power Button [PWRF] [ 2.928894] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.935418] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.949627] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.960944] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.980760] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.008484] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.035747] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.039549] Non-volatile memory driver v1.3 [ 3.040848] Linux agpgart interface v0.103 [ 3.075468] virtio_blk virtio1: [vda] 146608 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.078625] vda: detected capacity change from 0 to 75063296 [ 3.096943] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.099690] vdb: detected capacity change from 0 to 1073741824 [ 3.114086] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.117123] vdc: detected capacity change from 0 to 2621440000 [ 3.132188] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.135245] vdd: detected capacity change from 0 to 2621440000 [ 3.150816] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.154178] vde: detected capacity change from 0 to 4294967296 [ 3.171373] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.175339] vdf: detected capacity change from 0 to 4294967296 [ 3.181715] libphy: Fixed MDIO Bus: probed [ 3.187126] usbcore: registered new interface driver usbserial_generic [ 3.189080] usbserial: USB Serial support registered for generic [ 3.191693] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.196189] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.198263] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.200830] mousedev: PS/2 mouse device common for all mice [ 3.203895] rtc_cmos 00:05: RTC can wake from S4 [ 3.206745] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.207709] rtc_cmos 00:05: registered as rtc0 [ 3.212194] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.215334] intel_pstate: CPU model not supported [ 3.220111] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.223424] hid: raw HID events driver (C) Jiri Kosina [ 3.226162] usbcore: registered new interface driver usbhid [ 3.228193] usbhid: USB HID core driver [ 3.230089] drop_monitor: Initializing network drop monitor service [ 3.230196] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.232553] Initializing XFRM netlink socket [ 3.238194] NET: Registered protocol family 10 [ 3.241561] Segment Routing with IPv6 [ 3.242695] NET: Registered protocol family 17 [ 3.244936] mpls_gso: MPLS GSO support [ 3.250743] RAS: Correctable Errors collector initialized. [ 3.252736] AVX version of gcm_enc/dec engaged. [ 3.255093] AES CTR mode by8 optimization enabled [ 3.350312] sched_clock: Marking stable (3350289620, 0)->(4277658251, -927368631) [ 3.352706] registered taskstats version 1 [ 3.353914] Loading compiled-in X.509 certificates [ 3.355180] zswap: loaded using pool lzo/zbud [ 3.378362] Key type big_key registered [ 3.390102] Key type encrypted registered [ 3.392205] ima: No TPM chip found, activating TPM-bypass! [ 3.394742] ima: Allocated hash algorithm: sha1 [ 3.396900] ima: No architecture policies found [ 3.398914] evm: Initialising EVM extended attributes: [ 3.401110] evm: security.selinux [ 3.402467] evm: security.ima [ 3.403824] evm: security.capability [ 3.405148] evm: HMAC attrs: 0x1 [ 3.407778] rtc_cmos 00:05: setting system clock to 2026-08-27 02:42:17 UTC (1787798537) [ 3.414919] debug: unmapping init [mem 0xffffffffa0403000-0xffffffffa05fffff] [ 3.418367] debug: unmapping init [mem 0xffffffff9f182000-0xffffffff9f458fff] [ 3.431156] Write protecting the kernel read-only data: 28672k [ 3.434861] debug: unmapping init [mem 0xffffffff9d803000-0xffffffff9d9fffff] [ 3.437982] debug: unmapping init [mem 0xffffffff9e114000-0xffffffff9e1fffff] [ 3.490360] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.500101] systemd[1]: Detected virtualization kvm. [ 3.502318] systemd[1]: Detected architecture x86-64. [ 3.504423] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.537994] systemd[1]: No hostname configured. [ 3.540085] systemd[1]: Set hostname to . [ 3.542431] random: systemd: uninitialized urandom read (16 bytes read) [ 3.545224] systemd[1]: Initializing machine ID from random generator. [ 3.581591] random: ln: uninitialized urandom read (6 bytes read) [ 3.691850] random: systemd: uninitialized urandom read (16 bytes read) [ 3.694706] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.698703] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.703243] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.261175] device-mapper: uevent: version 1.0.3 [ 4.263119] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.916885] random: fast init done [ 4.939752] virtio_net virtio0 ens2: renamed from eth0 [ 4.971626] scsi host0: ata_piix [ 4.989265] scsi host1: ata_piix [ 4.991083] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.993398] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.454831] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.910293] random: crng init done [ 9.913746] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.355668] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.526188] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.733459] SELinux: Disabled at runtime. [ 11.793537] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.799340] systemd[1]: Detected virtualization kvm. [ 11.800771] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.293409] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.296654] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.301242] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.304277] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.306654] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.315731] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.326921] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ 12.410254] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.722631] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.057405] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.063163] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.173361] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.186413] EDAC sbridge: Ver: 1.1.2 [ 14.698740] Key type dns_resolver registered [ 14.976418] NFS: Registering the id_resolver key type [ 14.978440] Key type id_resolver registered [ 14.979907] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg119-server login: [ 41.107434] libcfs: loading out-of-tree module taints kernel. [ 41.123815] Key type ._llcrypt registered [ 41.124946] Key type .llcrypt registered [ 41.173202] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 50.207092] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 50.988359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.996814] alg: No test for adler32 (adler32-zlib) [ 52.080627] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 52.451675] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 54.088219] Key type lgssc registered [ 54.844373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 70.968335] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 103.738896] hrtimer: interrupt took 5532362 ns [ 118.339510] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 131.057431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 131.106355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 132.362530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 132.405577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 132.523977] Lustre: lustre-MDT0000: new disk, initializing [ 132.598676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 132.616697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 136.788570] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 150.771322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.932949] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 150.959662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 150.964681] Lustre: Skipped 1 previous similar message [ 151.069279] Lustre: lustre-MDT0001: new disk, initializing [ 151.143411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 151.163924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 151.176832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 155.335322] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.800880] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 168.616185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 168.965863] Lustre: lustre-OST0000: new disk, initializing [ 168.968790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 168.972782] Lustre: 8420:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 169.045989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 169.755112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 169.764030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 169.820510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 174.334706] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.467564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 188.666685] Lustre: lustre-OST0001: new disk, initializing [ 188.670533] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 188.674515] Lustre: 9491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 188.759128] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.647708] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.672473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 195.688919] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 195.804564] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 208.253491] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 213.939532] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 220.225731] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing check_logdir /tmp/testlogs/ [ 224.786945] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing yml_node [ 228.223235] Lustre: DEBUG MARKER: Client: 2.17.57.53 [ 230.371821] Lustre: DEBUG MARKER: MDS: 2.17.57.53 [ 232.402585] Lustre: DEBUG MARKER: OSS: 2.17.57.53 [ 233.609550] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 22:46:06 EDT 2026 [ 248.937342] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 250.918883] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 267.746561] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 267.749329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 267.756462] Lustre: Skipped 3 previous similar messages [ 267.780811] Lustre: Skipped 3 previous similar messages [ 274.417465] Lustre: server umount lustre-MDT0000 complete [ 277.987571] LustreError: 6494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 277.997492] LustreError: 6494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 278.027344] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798812 with bad export cookie 4630286232383076463 [ 278.033659] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 283.105374] LustreError: 6489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 283.119699] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.138148] LustreError: 6489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 283.183270] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 283.191458] Lustre: Skipped 1 previous similar message [ 284.620778] Lustre: server umount lustre-MDT0001 complete [ 288.919299] Lustre: server umount lustre-OST0000 complete [ 292.488362] Lustre: server umount lustre-OST0001 complete [ 299.165927] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 308.829263] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 344.005646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 344.206541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 344.227033] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 344.337967] Lustre: lustre-MDT0000: new disk, initializing [ 344.436908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 344.464424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 347.930222] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 357.990795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 358.091886] Lustre: 15046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 358.133801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 358.139357] Lustre: Skipped 1 previous similar message [ 358.212766] Lustre: lustre-MDT0001: new disk, initializing [ 358.340096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 358.419869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 358.435865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 362.943705] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 372.081475] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 378.162267] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 386.408907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 386.775282] Lustre: lustre-OST0000: new disk, initializing [ 386.785466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 386.793740] Lustre: 17017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 386.920777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.326769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 388.336749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 388.545031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 395.105627] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 408.467320] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 414.790585] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 414.968970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 419.760722] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 419.984175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 424.425088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 424.427511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.443169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 424.446940] Lustre: Skipped 2 previous similar messages [ 424.464097] Lustre: Skipped 1 previous similar message [ 428.874562] Lustre: server umount lustre-OST0000 complete [ 434.657639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.665759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 434.670550] Lustre: Skipped 1 previous similar message [ 434.689378] Lustre: Skipped 1 previous similar message [ 438.707206] Lustre: server umount lustre-MDT0000 complete [ 439.777389] LustreError: 15052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 439.789028] LustreError: 15052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 441.905760] LustreError: 15038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798975 with bad export cookie 4630286232383078003 [ 441.907282] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 441.915339] LustreError: 15038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 442.302869] Lustre: server umount lustre-MDT0001 complete [ 447.921154] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:49:40 (1787798980) [ 464.843288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.340780] LustreError: 19891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.426632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.910082] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 470.498104] LustreError: 19892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 478.818347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.317795] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 492.018915] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 497.787501] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 500.268852] Lustre: Failing over lustre-MDT0000 [ 500.425871] Lustre: server umount lustre-MDT0000 complete [ 500.707757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.708912] LustreError: 19887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.730775] Lustre: Skipped 1 previous similar message [ 500.746581] LustreError: 19887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 518.988940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.134762] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 524.521697] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 524.524391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 524.555551] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 524.803754] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 533.191575] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 540.129582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 540.134596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.142046] LustreError: Skipped 1 previous similar message [ 540.145338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 540.162514] Lustre: Skipped 1 previous similar message [ 543.865445] Lustre: server umount lustre-MDT0000 complete [ 545.249852] LustreError: 19888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.267621] LustreError: 19888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 547.525302] LustreError: 19872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799081 with bad export cookie 4630286232383079417 [ 547.538100] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.541703] LustreError: 19872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 547.830174] Lustre: server umount lustre-MDT0001 complete [ 556.093909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.474558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.477814] Lustre: Skipped 2 previous similar messages [ 560.460349] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 569.182033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.261152] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 581.275414] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 586.670733] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 588.712923] Lustre: Failing over lustre-MDT0000 [ 589.266523] Lustre: server umount lustre-MDT0000 complete [ 590.304801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 590.316632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.332479] LustreError: 23227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.350966] LustreError: 23227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 607.718719] Lustre: 3616:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787799125/real 1787799125] req@ffff9a6dc3abe580 x1874642690296192/t0(0) o400->MGC192.168.201.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787799141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 607.758558] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.609231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 617.333510] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 622.167489] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 622.565135] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 622.584539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 622.592332] Lustre: Skipped 1 previous similar message [ 622.606184] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 632.219093] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 637.922334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.929076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 637.938372] Lustre: Skipped 2 previous similar messages [ 637.968752] Lustre: Skipped 2 previous similar messages [ 642.963054] Lustre: server umount lustre-MDT0000 complete [ 647.256744] LustreError: 23208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799181 with bad export cookie 4630286232383080355 [ 647.271872] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 647.280332] LustreError: 23208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 647.815194] Lustre: server umount lustre-MDT0001 complete [ 657.337057] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:53:10 (1787799190) [ 674.786110] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 686.528722] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 728.595824] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 739.423637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 739.757192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 739.804697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 739.897888] Lustre: lustre-MDT0000: new disk, initializing [ 740.007426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 740.015529] Lustre: Skipped 2 previous similar messages [ 740.042537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 744.927189] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 756.558873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 756.762383] Lustre: 29604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 757.107325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 762.415593] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 772.736758] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 778.725436] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 786.099569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 786.378848] Lustre: 31571:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 787.787744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 787.822601] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 787.876699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 787.898081] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 793.150649] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 803.926289] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 809.854861] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 834.028670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 834.037521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 834.047756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 834.047765] Lustre: Skipped 2 previous similar messages [ 834.049634] LustreError: Skipped 1 previous similar message [ 834.103679] Lustre: Skipped 1 previous similar message [ 836.764757] Lustre: server umount lustre-OST0000 complete [ 844.269104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 846.923359] Lustre: server umount lustre-MDT0000 complete [ 849.380705] LustreError: 29615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.413548] LustreError: 29615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 850.941786] LustreError: 29597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799385 with bad export cookie 4630286232383080761 [ 850.957624] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 851.413259] Lustre: server umount lustre-MDT0001 complete [ 863.357816] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 866.755972] Key type lgssc unregistered [ 867.018923] LNet: 33550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.042345] LNetError: 33550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.063375] LNet: Removed LNI 192.168.201.119@tcp [ 868.106418] Key type .llcrypt unregistered [ 868.109447] Key type ._llcrypt unregistered [ 894.567188] Key type ._llcrypt registered [ 894.569429] Key type .llcrypt registered [ 894.656814] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 896.132284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 896.175810] alg: No test for adler32 (adler32-zlib) [ 897.276914] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 897.634337] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 899.297091] Key type lgssc registered [ 900.298836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 908.335700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 908.349357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.759056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 914.789819] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 919.023285] LustreError: 34967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 919.038850] LustreError: 34967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 924.139511] LustreError: 34966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 924.884497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.265304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 929.881102] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 938.157148] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 944.263949] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 951.210613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 951.607320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.709628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 957.501098] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 965.317539] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 970.045918] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 983.656490] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:58:36 (1787799516) [ 986.169427] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 988.848891] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 997.368957] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:58:49 (1787799529) [ 1010.244621] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:59:01 (1787799541) [ 1011.472167] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1022.926929] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1022.932655] Lustre: Skipped 2 previous similar messages [ 1024.795497] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:59:17 (1787799557) [ 1027.019963] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1027.031272] Lustre: Skipped 2 previous similar messages [ 1035.508168] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1035.512458] Lustre: Skipped 4 previous similar messages [ 1048.990643] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:59:41 (1787799581) [ 1051.257713] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1054.404578] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1062.842625] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:59:55 (1787799595) [ 1064.749854] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1064.753581] Lustre: Skipped 6 previous similar messages [ 1078.177357] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:00:10 (1787799610) [ 1098.210331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1098.212874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.221047] LustreError: Skipped 1 previous similar message [ 1098.221827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1098.230138] Lustre: Skipped 1 previous similar message [ 1098.243408] Lustre: Skipped 1 previous similar message [ 1099.677850] Lustre: server umount lustre-OST0000 complete [ 1103.334043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.337610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.344421] Lustre: Skipped 1 previous similar message [ 1103.356837] Lustre: Skipped 1 previous similar message [ 1108.450599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1109.595698] Lustre: server umount lustre-MDT0000 complete [ 1112.879353] LustreError: 38530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799646 with bad export cookie 15416140686002616896 [ 1112.883948] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1112.891123] LustreError: 38530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1113.288479] Lustre: server umount lustre-MDT0001 complete [ 1126.855494] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 1130.170708] Key type lgssc unregistered [ 1130.421300] LNet: 42844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1130.425314] LNetError: 42844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1130.444697] LNet: Removed LNI 192.168.201.119@tcp [ 1131.286104] Key type .llcrypt unregistered [ 1131.287414] Key type ._llcrypt unregistered [ 1149.211941] Key type ._llcrypt registered [ 1149.214904] Key type .llcrypt registered [ 1149.311256] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 1164.104877] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1165.104924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1165.117231] alg: No test for adler32 (adler32-zlib) [ 1166.154177] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 1166.360108] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1168.000463] Key type lgssc registered [ 1168.834020] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1202.594063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1202.638295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.876379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1203.898239] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1203.960358] Lustre: lustre-MDT0000: new disk, initializing [ 1204.016255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1204.029976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1208.648665] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1219.727151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1219.852579] Lustre: 45998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1219.924351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1219.927056] Lustre: Skipped 1 previous similar message [ 1220.020588] Lustre: lustre-MDT0001: new disk, initializing [ 1220.118546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1220.149321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1220.158738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1225.547257] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1235.528492] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1243.069520] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1250.668084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1251.046552] Lustre: lustre-OST0000: new disk, initializing [ 1251.051449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1251.061561] Lustre: 47963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1251.190775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1259.422993] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1260.126655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1260.148517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1260.331219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1270.697823] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1275.681426] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1275.942323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1280.540650] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1280.780707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1286.115370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1286.133908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.150701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1291.261933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1291.267074] Lustre: Skipped 1 previous similar message [ 1296.356404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1296.363212] Lustre: Skipped 1 previous similar message [ 1297.376150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1297.597432] Lustre: server umount lustre-OST0000 complete [ 1301.475115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.478119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1301.492199] Lustre: Skipped 2 previous similar messages [ 1301.501359] Lustre: Skipped 1 previous similar message [ 1306.594579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1306.601197] Lustre: Skipped 1 previous similar message [ 1307.616464] Lustre: server umount lustre-MDT0000 complete [ 1312.402769] LustreError: 45989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799846 with bad export cookie 3648460146198494910 [ 1312.410055] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1312.411684] LustreError: 45989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1312.687741] Lustre: server umount lustre-MDT0001 complete [ 1322.882262] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 23:04:15 (1787799855) [ 1330.993507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.450953] LustreError: 50432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1331.473969] LustreError: 50432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1331.539554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.308599] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1336.802890] LustreError: 50433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.922709] LustreError: 50432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1344.344354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.769577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1349.053456] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1356.538518] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1362.562611] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1369.566299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1369.779968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.631688] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1384.160616] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1392.834946] LustreError: 52960:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1393.822595] Lustre: Modifying parameter general.timeout=1 in log params [ 1394.838528] Lustre: Modifying parameter general.timeout=2 in log params [ 1397.569382] Lustre: Modifying parameter general.timeout=5 in log params [ 1397.572309] Lustre: Skipped 2 previous similar messages [ 1401.974378] Lustre: Modifying parameter general.timeout=9 in log params [ 1401.982131] Lustre: Skipped 3 previous similar messages [ 1410.754498] Lustre: Modifying parameter general.timeout=18 in log params [ 1410.757367] Lustre: Skipped 8 previous similar messages [ 1427.914416] Lustre: Modifying parameter general.timeout=35 in log params [ 1427.920231] Lustre: Skipped 16 previous similar messages [ 1460.732969] Lustre: Modifying parameter general.timeout=70 in log params [ 1460.740535] Lustre: Skipped 34 previous similar messages [ 1527.186233] Lustre: Disabling parameter general.timeout= in log params [ 1527.190654] Lustre: Skipped 59 previous similar messages [ 1530.401954] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 23:07:43 (1787800063) [ 1533.341554] Lustre: 59915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1535.390369] Lustre: 60013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1537.556584] Lustre: 60111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1541.988427] Lustre: 60319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1541.997945] Lustre: 60319:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1572.097067] Lustre: 61789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1572.105649] Lustre: 61789:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1572.133960] LustreError: 61789:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1576.309219] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 23:08:29 (1787800109) [ 1590.756804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1590.764072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1590.765963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.768617] Lustre: Skipped 1 previous similar message [ 1590.786336] LustreError: Skipped 1 previous similar message [ 1594.372602] Lustre: server umount lustre-MDT0000 complete [ 1595.880862] LustreError: 50432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.907461] LustreError: 50432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1597.521480] LustreError: 50413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800131 with bad export cookie 3648460146198495799 [ 1597.521736] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1597.529438] LustreError: 50413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1597.910413] Lustre: server umount lustre-MDT0001 complete [ 1611.325085] Lustre: server umount lustre-OST0000 complete [ 1641.145676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.362984] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1641.687895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.572924] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1653.979096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.152045] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1654.196707] Lustre: 64325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1658.422616] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1666.028146] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1671.642323] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1679.265607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1679.624709] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1679.993420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1680.005819] Lustre: Skipped 1 previous similar message [ 1687.524600] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1696.623409] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1698.905744] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1698.912316] Lustre: Skipped 23 previous similar messages [ 1698.979981] Lustre: 66890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1711.074219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1711.080687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.089716] Lustre: Skipped 1 previous similar message [ 1711.094799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1711.103485] Lustre: Skipped 2 previous similar messages [ 1712.956404] Lustre: server umount lustre-OST0000 complete [ 1721.313885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1721.319894] LustreError: Skipped 1 previous similar message [ 1721.330683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.349298] Lustre: Skipped 1 previous similar message [ 1725.295343] Lustre: server umount lustre-MDT0000 complete [ 1726.965752] LustreError: 64337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1726.982020] LustreError: 64337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1729.281262] LustreError: 64316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800263 with bad export cookie 3648460146198498942 [ 1729.284294] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1729.304354] LustreError: 64316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1729.739345] Lustre: server umount lustre-MDT0001 complete [ 1740.680415] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 1742.957749] Key type lgssc unregistered [ 1743.209795] LNet: 68100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.215699] LNetError: 68100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.226304] LNet: Removed LNI 192.168.201.119@tcp [ 1743.943841] Key type .llcrypt unregistered [ 1743.946951] Key type ._llcrypt unregistered [ 1773.259497] Key type ._llcrypt registered [ 1773.264132] Key type .llcrypt registered [ 1773.349369] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1774.459486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1774.543746] alg: No test for adler32 (adler32-zlib) [ 1775.582694] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 1775.849865] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1777.616179] Key type lgssc registered [ 1778.697409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1786.213168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1786.248965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1788.098721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.016384] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1798.632234] LustreError: 69514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1798.656179] LustreError: 69514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1802.336606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.110046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1808.544308] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1816.571771] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1822.575424] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1828.904683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1829.556090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.014475] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1842.409381] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1854.144797] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1862.635272] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.636608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1862.655205] Lustre: Skipped 2 previous similar messages [ 1862.674267] Lustre: Skipped 1 previous similar message [ 1867.750813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1867.777016] Lustre: Skipped 3 previous similar messages [ 1867.897800] Lustre: server umount lustre-MDT0000 complete [ 1871.798987] LustreError: 69493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800405 with bad export cookie 6786892080368996063 [ 1871.804310] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.806772] LustreError: 69493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1872.096299] Lustre: server umount lustre-MDT0001 complete [ 1885.393269] Lustre: server umount lustre-OST0000 complete [ 1915.281988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1915.692840] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1915.746607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1916.234575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1920.873322] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1930.102914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.174281] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1930.203587] Lustre: 74393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1930.882341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1935.725235] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1943.979065] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1949.310496] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1956.584283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1956.928278] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1957.585268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.074037] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1974.313874] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1976.391954] Lustre: 76972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1978.641963] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1986.532339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1986.542858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.577237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1987.043692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1990.738219] Lustre: server umount lustre-OST0000 complete [ 1997.282969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.283901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1997.285204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.285211] Lustre: Skipped 2 previous similar messages [ 1997.297254] Lustre: Skipped 2 previous similar messages [ 2000.680658] Lustre: server umount lustre-MDT0000 complete [ 2002.415221] LustreError: 75344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.433989] LustreError: 75344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2004.212878] LustreError: 74386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800538 with bad export cookie 6786892080368996903 [ 2004.213784] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.233419] LustreError: 74386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2004.617114] Lustre: server umount lustre-MDT0001 complete [ 2016.420799] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2019.162560] Key type lgssc unregistered [ 2019.494576] LNet: 78186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.506856] LNetError: 78186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.552794] LNet: Removed LNI 192.168.201.119@tcp [ 2020.481226] Key type .llcrypt unregistered [ 2020.484045] Key type ._llcrypt unregistered [ 2049.126733] Key type ._llcrypt registered [ 2049.128855] Key type .llcrypt registered [ 2049.224856] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2050.404589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2050.493319] alg: No test for adler32 (adler32-zlib) [ 2051.584143] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 2051.866605] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2053.544501] Key type lgssc registered [ 2054.779238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2063.319904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2063.356952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.421197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.100259] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2075.632773] LustreError: 79601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.644458] LustreError: 79601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2080.645133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.740213] LustreError: 79602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2081.519912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2085.807804] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2094.307353] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2101.255902] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2108.904848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2109.821742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.474520] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2125.039581] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2131.019541] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2131.850493] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2133.596190] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2133.608193] Lustre: Skipped 1 previous similar message [ 2136.294297] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2136.298372] Lustre: Skipped 2 previous similar messages [ 2140.510801] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2140.515334] Lustre: Skipped 3 previous similar messages [ 2149.565386] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2149.583116] Lustre: Skipped 6 previous similar messages [ 2165.968246] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2165.975357] Lustre: Skipped 13 previous similar messages [ 2198.841155] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2198.847433] Lustre: Skipped 31 previous similar messages [ 2243.271573] Lustre: 87713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2246.601940] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 23:19:39 (1787800779) [ 2248.570760] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2250.929404] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 23:19:43 (1787800783) [ 2259.433263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2259.436387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.442182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2259.459206] Lustre: Skipped 1 previous similar message [ 2259.483492] Lustre: Skipped 1 previous similar message [ 2262.448048] Lustre: server umount lustre-OST0000 complete [ 2269.666373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2269.674513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.680525] Lustre: Skipped 1 previous similar message [ 2269.689444] Lustre: Skipped 1 previous similar message [ 2272.757103] Lustre: server umount lustre-MDT0000 complete [ 2274.789935] LustreError: 79602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.818373] LustreError: 79602:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2277.034848] LustreError: 79583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800811 with bad export cookie 3932836969188832980 [ 2277.037547] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.043311] LustreError: 79583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2277.384683] Lustre: server umount lustre-MDT0001 complete [ 2290.901094] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2294.114836] Key type lgssc unregistered [ 2294.490529] LNet: 88965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.498408] LNetError: 88965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.532415] LNet: Removed LNI 192.168.201.119@tcp [ 2296.058404] Key type .llcrypt unregistered [ 2296.069664] Key type ._llcrypt unregistered [ 2301.899203] Key type ._llcrypt registered [ 2301.903611] Key type .llcrypt registered [ 2302.089407] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2307.142263] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules [ 2308.677172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2308.772433] alg: No test for adler32 (adler32-zlib) [ 2309.973935] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 2310.186940] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2311.976257] Key type lgssc registered [ 2313.202679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2314.170045] LustreError: 90169:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2336.270693] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2336.303333] LustreError: 90169:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2348.977750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2349.023198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.662568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2356.501345] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2360.807696] LustreError: 91194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.836526] LustreError: 91194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2365.926317] LustreError: 91193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.730234] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 23:21:39 (1787800899) [ 2371.048960] LustreError: 91194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.097067] LustreError: 91225:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2372.108935] LustreError: 91225:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2372.117947] LustreError: 91225:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2372.689445] Lustre: server umount lustre-MDT0000 complete [ 2389.170551] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2392.510961] Key type lgssc unregistered [ 2392.843637] LNet: 92722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2392.852899] LNetError: 92722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2392.872884] LNet: Removed LNI 192.168.201.119@tcp [ 2393.829468] Key type .llcrypt unregistered [ 2393.833798] Key type ._llcrypt unregistered [ 2413.400605] Key type ._llcrypt registered [ 2413.403311] Key type .llcrypt registered [ 2413.559233] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2414.580513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2414.632210] alg: No test for adler32 (adler32-zlib) [ 2415.900318] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 2416.107868] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2417.760351] Key type lgssc registered [ 2419.247367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2428.577399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2428.594439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2430.201708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.869285] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2440.679962] LustreError: 93940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.714218] LustreError: 93940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2444.759931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2445.290736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2450.120149] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2458.965286] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2465.214541] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2472.854128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2473.464489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.115899] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2488.738725] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2539.896899] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:24:32 (1787801072) [ 2542.197461] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2544.122940] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:24:36 (1787801076) [ 2552.297979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2552.305349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2552.329697] Lustre: Skipped 1 previous similar message [ 2555.285882] Lustre: server umount lustre-MDT0000 complete [ 2557.412978] LustreError: 93940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.433811] LustreError: 93940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2559.101199] LustreError: 93922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801093 with bad export cookie 1177679041473652549 [ 2559.122493] LustreError: 93922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2559.133307] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2559.551934] Lustre: server umount lustre-MDT0001 complete [ 2573.870466] Lustre: server umount lustre-OST0000 complete [ 2586.700050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.266244] LustreError: 98217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.456360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2591.858534] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2592.738411] LustreError: 98218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.861782] LustreError: 98217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.305352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.973565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2607.555404] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2617.490664] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2625.961703] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2641.411124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2641.688875] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2641.702140] LustreError: 98206:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2641.733595] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2641.743691] LustreError: 100303:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2641.750118] LustreError: 100303:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2641.756641] LustreError: 100303:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2641.815524] Lustre: server umount lustre-OST0000 complete [ 2641.819189] LustreError: 100303:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2652.022529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.359489] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2652.368251] LustreError: 98205:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2652.402429] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2652.411469] LustreError: 100770:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2652.419506] LustreError: 100770:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2652.430990] LustreError: 100770:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2652.584167] Lustre: server umount lustre-OST0000 complete [ 2652.592176] LustreError: 100770:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2663.020685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2663.353177] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2663.358319] Lustre: Found index 0 for lustre-OST0000, updating log [ 2663.365697] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2663.468625] Lustre: lustre-OST0000: new disk, initializing [ 2663.475851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2663.486214] Lustre: 101304:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2663.905893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2671.462718] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2676.929574] Lustre: Failing over lustre-OST0000 [ 2677.191955] Lustre: server umount lustre-OST0000 complete [ 2683.873488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2683.877491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.895412] Lustre: Skipped 1 previous similar message [ 2683.903786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2683.912031] Lustre: Skipped 2 previous similar messages [ 2686.434523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2686.436780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2686.472675] Lustre: Skipped 1 previous similar message [ 2688.391649] Lustre: server umount lustre-MDT0000 complete [ 2691.562070] LustreError: 99164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.586802] LustreError: 99164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2692.698447] LustreError: 102066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801226 with bad export cookie 1177679041473655293 [ 2692.703888] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2692.715796] LustreError: 102066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2702.508251] Lustre: server umount lustre-MDT0001 complete [ 2711.985488] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:27:24 (1787801244) [ 2731.020903] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2740.908167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.364503] LustreError: 103653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.456395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.312329] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2753.460398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.038199] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2762.203391] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2769.366744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2770.243344] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2777.633150] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2778.409877] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2778.453293] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2778.453897] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2786.994955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2787.105096] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2787.186607] Lustre: lustre-OST0001: new disk, initializing [ 2787.190901] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2787.201816] Lustre: 105975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2787.510361] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2787.519850] Lustre: Skipped 2 previous similar messages [ 2795.301194] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2796.593219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2796.604244] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2796.789602] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2804.484341] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2813.477156] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2819.066708] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.19@tcp (stopping) [ 2821.107811] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2821.113607] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.152592] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.19@tcp (stopping) [ 2824.158687] Lustre: Skipped 4 previous similar messages [ 2832.353066] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2832.609853] Lustre: server umount lustre-MDT0001 complete [ 2832.875109] LustreError: 105172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.894228] LustreError: 105172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2843.212853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.297761] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2843.308274] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2843.365579] Lustre: 103643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2844.653797] LustreError: 107513:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2844.665853] LustreError: 107513:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2844.676284] LustreError: 103645:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2844.794762] LustreError: 107519:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.119@tcp: cfg command failed: rc = -17 [ 2844.809648] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2844.809648] [ 2849.272492] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2849.632639] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2859.483880] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 23:29:52 (1787801392) [ 2877.296917] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2895.412991] Lustre: 109588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2918.894789] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2923.749595] Lustre: 110724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3013.090167] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3013.099722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3013.101494] Lustre: Skipped 5 previous similar messages [ 3013.111316] Lustre: Skipped 7 previous similar messages [ 3019.341081] Lustre: server umount lustre-MDT0000 complete [ 3022.716147] LustreError: 103634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801556 with bad export cookie 1177679041473656294 [ 3022.720469] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3022.730815] LustreError: 103634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3023.121583] Lustre: server umount lustre-MDT0001 complete [ 3037.851281] Lustre: server umount lustre-OST0000 complete [ 3039.584430] Lustre: 93253:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787801557/real 1787801557] req@ffff9a6dc2f68f00 x1874645169316992/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787801573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3039.609306] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.303357] Lustre: server umount lustre-OST0001 complete [ 3046.478476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3056.479551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3064.804196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3075.645875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3096.521758] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3105.976637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.995146] Lustre: lustre-MDT0000: reset Object Index mappings [ 3106.611030] LustreError: 114678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3106.627991] LustreError: 114678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3106.769514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3106.775688] Lustre: Skipped 1 previous similar message [ 3111.546428] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3120.427374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.443866] Lustre: lustre-MDT0001: reset Object Index mappings [ 3125.641746] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3128.152600] Lustre: 115827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3134.653549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3141.057410] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3145.521967] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3147.895321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3151.439143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3151.457463] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3151.458445] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3154.237804] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3162.150186] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3165.681032] Lustre: 117704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3210.361856] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 23:35:42 (1787801742) [ 3238.373882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.387755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3238.410907] Lustre: Skipped 3 previous similar messages [ 3238.428305] Lustre: Skipped 7 previous similar messages [ 3243.497179] LustreError: 114675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3243.516981] LustreError: 114675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3243.807093] Lustre: server umount lustre-MDT0000 complete [ 3247.733229] LustreError: 115429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787801781 with bad export cookie 1177679041473703187 [ 3247.748736] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3247.749745] LustreError: 115429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3254.295823] Lustre: server umount lustre-MDT0001 complete [ 3264.566374] Lustre: server umount lustre-OST0000 complete [ 3282.400173] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3282.527923] Lustre: server umount lustre-OST0001 complete [ 3289.382506] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 3298.636375] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3339.958381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.298266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3340.343798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3340.469299] Lustre: lustre-MDT0000: new disk, initializing [ 3340.548386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3340.560100] Lustre: Skipped 3 previous similar messages [ 3340.571504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3344.900254] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3352.412718] Lustre: server umount lustre-MDT0000 complete [ 3361.576075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.934581] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3361.950734] Lustre: 121974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3361.963880] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3366.648218] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3372.499542] Lustre: server umount lustre-MDT0000 complete [ 3382.381391] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:38:35 (1787801915) [ 3384.517785] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3386.683861] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:38:39 (1787801919) [ 3406.500033] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3417.982722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.554296] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3431.150821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3431.287879] Lustre: 124027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3431.341305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3431.345469] Lustre: Skipped 1 previous similar message [ 3431.415909] Lustre: lustre-MDT0001: new disk, initializing [ 3431.655254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3431.669169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3436.697451] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3441.662898] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3448.858897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3449.282530] Lustre: lustre-OST0000: new disk, initializing [ 3449.289484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3449.297233] Lustre: 125604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3450.721567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3450.740915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3451.039130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3456.299250] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3472.428883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3472.851963] Lustre: lustre-OST0001: new disk, initializing [ 3472.866989] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3472.890503] Lustre: 126477:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3474.060682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3474.087053] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3474.335764] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3483.109822] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3495.408840] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3500.607408] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3893.228684] Lustre: 132889:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3893.238090] Lustre: 132889:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3893.243523] Lustre: 132889:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3893.253732] Lustre: 132889:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3893.264517] Lustre: 132889:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3893.271029] Lustre: 132889:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4170.612844] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:51:41 (1787802701) [ 4189.165701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4189.185878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4189.203115] Lustre: Skipped 5 previous similar messages [ 4189.230738] Lustre: Skipped 7 previous similar messages [ 4193.678380] Lustre: server umount lustre-MDT0000 complete [ 4194.275851] LustreError: 127282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.300095] LustreError: 127282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4199.205409] LustreError: 124019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787802733 with bad export cookie 1177679041473769512 [ 4199.228143] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4199.232709] LustreError: 124019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4199.399995] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4199.426159] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4199.429809] Lustre: Skipped 1 previous similar message [ 4206.630165] Lustre: server umount lustre-MDT0001 complete [ 4217.290383] Lustre: server umount lustre-OST0000 complete [ 4227.625453] Lustre: server umount lustre-OST0001 complete [ 4234.727148] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 4246.823818] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 4287.404605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.729577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4287.785967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4287.929613] Lustre: lustre-MDT0000: new disk, initializing [ 4288.032886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.047081] Lustre: Skipped 5 previous similar messages [ 4288.071291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4292.415908] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4303.450644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.625392] Lustre: 135939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4303.691181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4303.693565] Lustre: Skipped 1 previous similar message [ 4303.756741] Lustre: lustre-MDT0001: new disk, initializing [ 4303.872789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4303.893324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4307.858814] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4318.055471] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4325.363275] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4334.635872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4335.071491] Lustre: lustre-OST0000: new disk, initializing [ 4335.083247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4335.101110] Lustre: 137910:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4335.276832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4335.285708] Lustre: Skipped 1 previous similar message [ 4337.059641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4337.084986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4337.217129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4343.487865] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4353.883261] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4363.828323] Lustre: lustre-MDD0000: changelog on [ 4366.635862] Lustre: lustre-MDD0001: changelog on [ 6327.105831] Lustre: lustre-MDD0001: changelog off [ 6329.572314] Lustre: 142974:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6330.678507] Lustre: lustre-MDD0000: changelog off [ 6339.043378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6339.045417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6339.059167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6339.070343] Lustre: Skipped 2 previous similar messages [ 6339.078438] Lustre: Skipped 2 previous similar messages [ 6343.394065] Lustre: server umount lustre-OST0000 complete [ 6349.291299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6349.315814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6349.318828] Lustre: Skipped 2 previous similar messages [ 6354.404493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6354.412467] Lustre: Skipped 1 previous similar message [ 6359.523867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6359.530806] Lustre: Skipped 1 previous similar message [ 6362.080179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6362.387513] Lustre: server umount lustre-MDT0000 complete [ 6364.642688] LustreError: 136810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6364.671737] LustreError: 136810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6365.613035] LustreError: 135933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787804899 with bad export cookie 1177679041474546596 [ 6365.615471] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6365.626829] LustreError: 135933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6366.074409] Lustre: server umount lustre-MDT0001 complete [ 6376.863316] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 6379.290468] Key type lgssc unregistered [ 6379.525832] LNet: 147078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6379.529903] LNetError: 147078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6380.584510] LNet: Removed LNI 192.168.201.119@tcp [ 6381.650319] Key type .llcrypt unregistered [ 6381.655047] Key type ._llcrypt unregistered [ 6402.870419] Key type ._llcrypt registered [ 6402.872811] Key type .llcrypt registered [ 6402.970514] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 6417.048072] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 6418.282097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6418.320748] alg: No test for adler32 (adler32-zlib) [ 6419.417467] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 6419.729399] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 6421.432365] Key type lgssc registered [ 6422.500326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6460.772718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6460.813291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6462.210192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6462.257531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6462.348928] Lustre: lustre-MDT0000: new disk, initializing [ 6462.470036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6462.518748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6467.792397] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6478.836799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6478.950843] Lustre: 150282:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6479.115143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6479.122545] Lustre: Skipped 1 previous similar message [ 6479.327842] Lustre: lustre-MDT0001: new disk, initializing [ 6479.542930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6479.688876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6479.708435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6485.093631] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6497.001988] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6503.537797] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6510.666925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6511.032753] Lustre: lustre-OST0000: new disk, initializing [ 6511.036501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6511.043853] Lustre: 152253:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6511.146869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6517.007926] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6518.332456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6518.349166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6518.492992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6526.772921] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6533.489865] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_module obdecho/obdecho [ 6535.542855] LustreError: 153255:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6535.553893] LustreError: 153255:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6537.275450] LustreError: 153352:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6549.487392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6549.496419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6549.499976] Lustre: Skipped 2 previous similar messages [ 6549.509786] Lustre: Skipped 2 previous similar messages [ 6554.242729] Lustre: server umount lustre-MDT0000 complete [ 6558.053747] LustreError: 150272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805092 with bad export cookie 5867194917389322652 [ 6558.070264] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6558.072443] LustreError: 150272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6558.504840] Lustre: server umount lustre-MDT0001 complete [ 6572.774213] Lustre: server umount lustre-OST0000 complete [ 6596.888126] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 6611.621854] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 6651.466752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6651.813862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6651.884672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6651.978893] Lustre: lustre-MDT0000: new disk, initializing [ 6652.054524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6652.078354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6657.276890] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6668.884726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6668.975605] Lustre: 157102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6669.047954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6669.051355] Lustre: Skipped 1 previous similar message [ 6669.199077] Lustre: lustre-MDT0001: new disk, initializing [ 6669.349109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6669.426874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6669.439080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6675.240434] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6685.474527] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6691.976907] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6699.283186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6699.677431] Lustre: lustre-OST0000: new disk, initializing [ 6699.681547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6699.686721] Lustre: 159069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6699.790428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6705.733501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6705.742295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6705.857310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6706.000872] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6715.290664] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6720.202619] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6720.411563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6725.082057] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6725.274675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6731.752945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6731.763055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6731.794923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6731.803350] Lustre: Skipped 1 previous similar message [ 6733.468989] Lustre: server umount lustre-OST0000 complete [ 6740.961925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6740.966979] LustreError: Skipped 1 previous similar message [ 6740.970451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6740.977670] Lustre: Skipped 1 previous similar message [ 6740.985622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6743.685674] Lustre: server umount lustre-MDT0000 complete [ 6747.106305] LustreError: 158175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6747.125520] LustreError: 158175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6747.549360] LustreError: 157095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805281 with bad export cookie 5867194917389323611 [ 6747.560300] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6747.566682] LustreError: 157095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6748.398816] Lustre: server umount lustre-MDT0001 complete [ 6751.633430] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 00:34:44 (1787805284) [ 6753.134361] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6754.683526] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 00:34:47 (1787805287) [ 6761.859454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6762.315196] LustreError: 161483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6762.389821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6766.313200] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6767.598126] LustreError: 161484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6772.707849] LustreError: 161483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6773.478897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6777.602980] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6784.212961] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6789.597549] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6796.854780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6797.133523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6797.139855] Lustre: Skipped 1 previous similar message [ 6802.987440] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6809.542577] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6817.766627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.769177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6817.774654] Lustre: Skipped 2 previous similar messages [ 6817.782903] Lustre: Skipped 4 previous similar messages [ 6821.285284] Lustre: server umount lustre-MDT0000 complete [ 6822.892217] LustreError: 162211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6822.908421] LustreError: 162211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6825.029292] LustreError: 161464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805359 with bad export cookie 5867194917389324500 [ 6825.038175] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6825.044715] LustreError: 161464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6825.442667] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.19@tcp (stopping) [ 6825.445880] Lustre: Skipped 1 previous similar message [ 6828.001823] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6828.007266] Lustre: Skipped 1 previous similar message [ 6831.012756] LustreError: 163325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6831.028873] LustreError: 163325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6831.520319] Lustre: server umount lustre-MDT0001 complete [ 6836.320701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6845.903936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6846.111794] LustreError: 165323:0:(llog.c:1655:llog_backup()) MGC192.168.201.119@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6846.117263] Lustre: 165323:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.119@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6855.188720] LustreError: 165328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6890.966195] LustreError: 165328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6890.974946] LustreError: 165328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 30 previous similar messages [ 6907.360352] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6907.371395] Lustre: 165323:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.119@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6947.913954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6950.954063] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6951.088799] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6956.006500] LustreError: 163315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6956.030376] LustreError: 163315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 44 previous similar messages [ 6957.043790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6974.088903] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6975.980036] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6980.706533] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6981.634857] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6985.168431] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6986.310580] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7001.576954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7001.584342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7001.584856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7001.601080] Lustre: Skipped 5 previous similar messages [ 7005.950689] Lustre: server umount lustre-MDT0000 complete [ 7008.357907] LustreError: 161465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805542 with bad export cookie 5867194917389330961 [ 7008.364983] LustreError: 161465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7008.369609] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7008.705236] Lustre: server umount lustre-MDT0001 complete [ 7021.523689] Lustre: server umount lustre-OST0000 complete [ 7037.706780] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 7047.261674] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 7080.882769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7081.081544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7081.122757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7081.213993] Lustre: lustre-MDT0000: new disk, initializing [ 7081.265839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7081.269275] Lustre: Skipped 1 previous similar message [ 7081.276371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7085.120773] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7093.583390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7093.651106] Lustre: 170423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7093.695577] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7093.825114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7097.411413] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7105.652103] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7110.624303] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7116.528333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7116.761698] Lustre: lustre-OST0000: new disk, initializing [ 7116.766644] Lustre: Skipped 1 previous similar message [ 7116.770884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7116.776773] Lustre: Skipped 2 previous similar messages [ 7116.780273] Lustre: 172391:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7122.235711] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7125.587499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7125.600960] Lustre: Skipped 1 previous similar message [ 7125.611227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7125.752044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7131.506899] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7135.756842] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7135.882990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7139.711303] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7139.848388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7146.466441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7146.479743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7146.507802] Lustre: Skipped 2 previous similar messages [ 7146.518619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7147.779081] Lustre: server umount lustre-OST0000 complete [ 7157.897188] Lustre: server umount lustre-MDT0000 complete [ 7161.735769] LustreError: 170416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805695 with bad export cookie 5867194917389347789 [ 7161.755863] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7162.721613] Lustre: server umount lustre-MDT0001 complete [ 7172.688528] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:41:45 (1787805705) [ 7178.588373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7178.888564] LustreError: 174803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7178.907179] LustreError: 174803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 7183.614561] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7192.022279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7196.539889] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7205.270953] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7211.520936] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7219.343904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7227.427143] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7236.657865] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7246.760445] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:42:59 (1787805779) [ 7250.913906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7250.927452] LustreError: Skipped 1 previous similar message [ 7250.936591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7250.957018] Lustre: Skipped 3 previous similar messages [ 7250.965298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7250.975058] Lustre: Skipped 5 previous similar messages [ 7256.760272] Lustre: server umount lustre-OST0000 complete [ 7267.640668] Lustre: server umount lustre-MDT0000 complete [ 7271.370326] LustreError: 174783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805805 with bad export cookie 5867194917389348671 [ 7271.383768] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7271.385018] LustreError: 174783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7271.943305] Lustre: server umount lustre-MDT0001 complete [ 7283.907917] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 7286.862681] Key type lgssc unregistered [ 7287.278148] LNet: 178532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7287.286792] LNetError: 178532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7287.312695] LNet: Removed LNI 192.168.201.119@tcp [ 7288.404737] Key type .llcrypt unregistered [ 7288.411556] Key type ._llcrypt unregistered [ 7309.150768] Key type ._llcrypt registered [ 7309.152902] Key type .llcrypt registered [ 7309.307071] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 7311.155957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7311.183489] alg: No test for adler32 (adler32-zlib) [ 7312.340854] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 7312.663582] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 7314.368233] Key type lgssc registered [ 7315.637689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7323.854952] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7323.872110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7339.432485] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7344.544361] LustreError: 179780:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 7349.673927] LustreError: 179780:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7349.706445] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7349.723035] LustreError: MGC192.168.201.119@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7349.733566] LustreError: 179780:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7349.763681] LustreError: 179780:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7349.773650] LustreError: 179780:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7349.991196] Lustre: server umount lustre-OST0000 complete [ 7349.997338] LustreError: 179780:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7361.206764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7362.843559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7367.596361] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7372.260649] LustreError: 180282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7372.280545] LustreError: 180282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7376.842514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7377.126500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7381.625647] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7388.199036] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7394.790281] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7402.373104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7402.729353] LustreError: 182097:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7402.742888] Lustre: 182097:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.119@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7402.923336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7409.174590] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7417.575419] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7422.449386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7422.452146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7422.460227] Lustre: Skipped 2 previous similar messages [ 7422.479536] Lustre: Skipped 2 previous similar messages [ 7423.527345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7423.539306] Lustre: Skipped 2 previous similar messages [ 7426.066769] Lustre: server umount lustre-MDT0000 complete [ 7428.578428] LustreError: 180282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7428.593456] LustreError: 180282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7430.114013] LustreError: 180263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787805964 with bad export cookie 1973908811016783689 [ 7430.125206] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7430.126499] LustreError: 180263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7431.776691] LustreError: 181011:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003cc4ebc1 x1874650302804480/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7432.110726] Lustre: server umount lustre-MDT0001 complete [ 7446.744663] Lustre: server umount lustre-OST0000 complete [ 7455.653640] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 7464.461346] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 7505.664911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.879639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7505.910884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7505.982526] Lustre: lustre-MDT0000: new disk, initializing [ 7506.049484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7506.063538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7510.085692] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7519.771571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.894042] Lustre: 185624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7519.979640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7519.983936] Lustre: Skipped 1 previous similar message [ 7520.113924] Lustre: lustre-MDT0001: new disk, initializing [ 7520.174659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7520.202664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7520.207921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7524.445809] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7534.393793] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7540.038879] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7548.060163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7548.301540] Lustre: lustre-OST0000: new disk, initializing [ 7548.304734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7548.309970] Lustre: 187589:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7548.377929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7554.966872] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7557.714776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7557.727170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7557.878365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7564.131781] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7569.121821] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7569.324729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7573.531694] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7573.712136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7578.615169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7578.622967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.632207] LustreError: Skipped 1 previous similar message [ 7578.649828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7578.656991] Lustre: Skipped 1 previous similar message [ 7581.664643] Lustre: server umount lustre-OST0000 complete [ 7586.785433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7586.790111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7586.797889] Lustre: Skipped 1 previous similar message [ 7586.800892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7588.841094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7591.517977] Lustre: server umount lustre-MDT0000 complete [ 7593.962926] LustreError: 185629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7593.978058] LustreError: 185629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7596.220087] LustreError: 185614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806130 with bad export cookie 1973908811016784403 [ 7596.231062] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7596.244747] LustreError: 185614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7596.690871] Lustre: server umount lustre-MDT0001 complete [ 7607.363124] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:48:59 (1787806139) [ 7625.986663] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 7629.245489] Key type lgssc unregistered [ 7629.566404] LNet: 190540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7629.575661] LNetError: 190540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7630.635234] LNet: Removed LNI 192.168.201.119@tcp [ 7631.591960] Key type .llcrypt unregistered [ 7631.610045] Key type ._llcrypt unregistered [ 7654.176370] Key type ._llcrypt registered [ 7654.178124] Key type .llcrypt registered [ 7654.343241] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 7656.258627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7656.592890] alg: No test for adler32 (adler32-zlib) [ 7658.133820] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 7658.621434] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 7660.408155] Key type lgssc registered [ 7662.719710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7675.153083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7675.206587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7676.889755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7682.510723] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7686.116415] LustreError: 191760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7686.151468] LustreError: 191760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7691.249423] LustreError: 191759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7692.502194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7692.850465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7696.673306] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7706.516388] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7714.728602] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7727.249638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7727.652892] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7727.666305] LustreError: 193626:0:(llog.c:1696:llog_backup()) MGC192.168.201.119@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7727.681058] Lustre: 193626:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.119@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7727.913067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7736.926438] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7745.004764] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7749.089612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7749.100368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7749.119539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7751.657492] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7751.666724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7751.696121] Lustre: Skipped 1 previous similar message [ 7753.785593] Lustre: server umount lustre-MDT0000 complete [ 7756.775188] LustreError: 192950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7756.822848] LustreError: 192950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7757.562360] LustreError: 191741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806291 with bad export cookie 17323553710387488111 [ 7757.572204] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7757.580314] LustreError: 191741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7758.065748] Lustre: server umount lustre-MDT0001 complete [ 7772.798135] Lustre: server umount lustre-OST0000 complete [ 7784.652265] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 7796.598842] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 7837.883809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7838.281765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7838.337492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7838.395933] Lustre: lustre-MDT0000: new disk, initializing [ 7838.459114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7838.475344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7844.389590] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7859.295347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7859.448197] Lustre: 197152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7859.473913] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7859.476447] Lustre: Skipped 1 previous similar message [ 7859.477803] LustreError: 197935:0:(llog.c:1696:llog_backup()) MGC192.168.201.119@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7859.480504] LustreError: 197935:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7859.482561] Lustre: 197935:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.119@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7859.493246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7859.495848] Lustre: Skipped 1 previous similar message [ 7859.546279] Lustre: lustre-MDT0001: new disk, initializing [ 7859.663863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7859.697223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7859.700952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7864.555797] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7875.703358] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7882.069316] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7892.505441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7892.946817] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7892.949853] Lustre: Skipped 1 previous similar message [ 7892.953813] LustreError: 199101:0:(llog.c:1696:llog_backup()) MGC192.168.201.119@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7892.966453] LustreError: 199101:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7892.974950] Lustre: 199101:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.119@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7892.994445] Lustre: 199101:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7893.034258] Lustre: lustre-OST0000: new disk, initializing [ 7893.042244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7893.053108] Lustre: 199124:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7893.212183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7902.299741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7902.308163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7902.509150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7902.882473] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7913.853851] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7919.390496] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7919.622466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7923.836123] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7924.021102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7928.292923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7928.304182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7928.308337] Lustre: Skipped 2 previous similar messages [ 7928.308696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7928.332227] Lustre: Skipped 1 previous similar message [ 7932.365317] Lustre: server umount lustre-OST0000 complete [ 7936.481655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7936.487658] LustreError: Skipped 1 previous similar message [ 7936.492907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7936.508034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7942.582525] Lustre: server umount lustre-MDT0000 complete [ 7943.657841] LustreError: 198432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7943.690841] LustreError: 198432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7946.531126] LustreError: 197948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806480 with bad export cookie 17323553710387488832 [ 7946.535575] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7946.536863] LustreError: 197948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7946.844748] Lustre: server umount lustre-MDT0001 complete [ 7959.308754] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:54:51 (1787806491) [ 7979.323767] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 7992.255492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7992.950216] LustreError: 202253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7993.067749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7999.613791] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8003.575149] LustreError: 202254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8003.621256] LustreError: 202254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8013.640737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8022.169975] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8026.922177] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8034.913741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8035.560937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.580072] Lustre: Skipped 1 previous similar message [ 8043.332678] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8053.645783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8053.798938] Lustre: lustre-OST0001: new disk, initializing [ 8053.802280] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8053.808430] Lustre: 204561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8060.016463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8060.026587] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8060.172735] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8060.523747] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8072.303888] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8080.020872] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8082.118093] Lustre: DEBUG MARKER: ADD OST3 [ 8088.280086] loop: module loaded [ 8088.345517] loop0: detected capacity change from 0 to 204800000 [ 8088.569198] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8088.576993] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8088.582659] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8096.666215] Lustre: DEBUG MARKER: STOP OST3 [ 8097.291950] loop0: detected capacity change from 0 to 204800000 [ 8105.482575] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8105.687763] Lustre: lustre-OST0003: new disk, initializing [ 8105.692313] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8105.698926] Lustre: 206646:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8105.789497] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8105.794428] Lustre: Skipped 1 previous similar message [ 8107.195306] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8107.400333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8107.414268] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8107.455696] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8108.169146] Lustre: Failing over lustre-OST0003 [ 8108.438118] Lustre: server umount lustre-OST0003 complete [ 8109.216742] LustreError: 206773:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8112.626085] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8112.629653] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8112.636880] LustreError: 203739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8112.642148] Lustre: Skipped 2 previous similar messages [ 8112.670716] LustreError: 203739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8113.068713] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8126.285772] Lustre: DEBUG MARKER: START OST3 again [ 8134.285961] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8134.726810] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8136.116652] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8136.596434] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8136.599815] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8143.028464] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8168.545257] Lustre: Failing over lustre-OST0003 [ 8168.644394] Lustre: server umount lustre-OST0003 complete [ 8170.486764] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8170.506087] LustreError: 203740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8170.509813] Lustre: Skipped 1 previous similar message [ 8170.568465] LustreError: 203740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 8190.947524] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8190.954660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8190.967273] Lustre: Skipped 2 previous similar messages [ 8190.993442] Lustre: Skipped 5 previous similar messages [ 8194.746148] Lustre: server umount lustre-MDT0000 complete [ 8199.943826] LustreError: 202235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806734 with bad export cookie 17323553710387489714 [ 8199.948514] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8199.950361] LustreError: 202235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8201.193417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8201.218472] Lustre: Skipped 1 previous similar message [ 8206.807058] Lustre: server umount lustre-MDT0001 complete [ 8213.673453] Lustre: server umount lustre-OST0000 complete [ 8220.103437] Lustre: server umount lustre-OST0001 complete [ 8229.053700] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 8238.566808] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 8288.278639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8288.734361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8288.783904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8288.862921] Lustre: lustre-MDT0000: new disk, initializing [ 8288.950361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8288.953317] Lustre: Skipped 1 previous similar message [ 8288.967796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8294.529864] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8307.289488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8307.470238] Lustre: 212178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8307.546439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8307.550710] Lustre: Skipped 1 previous similar message [ 8307.663333] Lustre: lustre-MDT0001: new disk, initializing [ 8307.776142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8307.786918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8312.041613] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8323.683188] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8331.579802] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8341.757232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8342.213318] Lustre: lustre-OST0000: new disk, initializing [ 8342.220405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8342.228752] Lustre: 214158:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8343.932530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8343.937964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8344.037885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8349.561020] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8362.075826] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8367.765565] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8368.034700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8373.851670] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8374.165732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8379.887265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8379.889491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8379.895218] Lustre: Skipped 3 previous similar messages [ 8379.900349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8379.900357] Lustre: Skipped 2 previous similar messages [ 8391.136254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8391.540652] Lustre: server umount lustre-OST0000 complete [ 8399.842274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8404.107761] Lustre: server umount lustre-MDT0000 complete [ 8405.477762] LustreError: 215181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8405.508511] LustreError: 215181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8408.192654] LustreError: 212168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787806942 with bad export cookie 17323553710387492010 [ 8408.210407] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8408.219673] LustreError: 212168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8408.677113] Lustre: server umount lustre-MDT0001 complete [ 8412.954126] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:02:25 (1787806945) [ 8433.472170] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 8451.498842] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 8504.966458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8505.509697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8505.617206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8505.864188] Lustre: lustre-MDT0000: new disk, initializing [ 8506.157641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8510.515296] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8522.457615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8522.638540] Lustre: 218797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8522.957593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8528.260624] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8538.929055] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8546.563189] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8555.923604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8556.335973] Lustre: 220766:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8556.518084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8556.523692] Lustre: Skipped 4 previous similar messages [ 8557.709548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8557.796795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8563.919865] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8575.288788] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8581.627070] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8582.019583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8587.141449] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8587.333484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8593.890852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8593.893402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8593.897336] LustreError: Skipped 1 previous similar message [ 8593.906960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8593.914186] Lustre: Skipped 3 previous similar messages [ 8593.927094] Lustre: Skipped 9 previous similar messages [ 8595.772112] Lustre: server umount lustre-OST0000 complete [ 8614.370876] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8614.670920] Lustre: server umount lustre-MDT0000 complete [ 8619.183549] LustreError: 218788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787807153 with bad export cookie 17323553710387492892 [ 8619.198664] LustreError: 218788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8619.199285] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8619.625520] Lustre: server umount lustre-MDT0001 complete [ 8629.312969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8629.839882] LustreError: 222986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8634.169515] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8643.690312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8647.927888] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8656.560470] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8664.418835] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8673.525906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8681.914390] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8691.325700] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8705.001620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8705.001971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8705.013275] Lustre: Skipped 7 previous similar messages [ 8708.207638] Lustre: server umount lustre-OST0000 complete [ 8718.354032] Lustre: server umount lustre-MDT0000 complete [ 8722.408522] LustreError: 224831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787807256 with bad export cookie 17323553710387493774 [ 8722.415725] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8722.419260] LustreError: 224831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8722.946841] Lustre: server umount lustre-MDT0001 complete [ 8830.802277] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 8834.927400] Key type lgssc unregistered [ 8835.342262] LNet: 226657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8835.352732] LNetError: 226657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8835.381249] LNet: Removed LNI 192.168.201.119@tcp [ 8836.293268] Key type .llcrypt unregistered [ 8836.297976] Key type ._llcrypt unregistered [ 8864.090377] Key type ._llcrypt registered [ 8864.093616] Key type .llcrypt registered [ 8864.192756] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 8883.096362] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 8884.373661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8884.560511] alg: No test for adler32 (adler32-zlib) [ 8885.694324] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 8885.985477] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 8887.672699] Key type lgssc registered [ 8889.181129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8930.515432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8930.542398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8931.948692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8931.984742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8932.081475] Lustre: lustre-MDT0000: new disk, initializing [ 8932.126659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8932.146449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8937.269613] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8950.030811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8950.276984] Lustre: 230012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8950.340344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8950.346886] Lustre: Skipped 1 previous similar message [ 8950.466970] Lustre: lustre-MDT0001: new disk, initializing [ 8950.475495] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8950.609099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8950.667336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8950.682906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8956.981767] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8967.566499] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8974.512420] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8982.544653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8982.812855] Lustre: lustre-OST0000: new disk, initializing [ 8982.818767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8982.829948] Lustre: 231980:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8982.968723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8988.339329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8988.355422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8988.489492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8990.317479] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9001.396242] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9007.584176] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9007.835930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9013.023954] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9013.270993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9019.376064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9019.381767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9019.392917] LustreError: Skipped 1 previous similar message [ 9019.393954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9019.406461] Lustre: Skipped 1 previous similar message [ 9021.597436] Lustre: server umount lustre-OST0000 complete [ 9027.553702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9027.569113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9027.585750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9027.594678] Lustre: Skipped 1 previous similar message [ 9029.614811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9029.623277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9034.722098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9034.726736] Lustre: Skipped 2 previous similar messages [ 9040.353137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9040.742894] Lustre: server umount lustre-MDT0000 complete [ 9046.007192] LustreError: 230004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787807580 with bad export cookie 13936684680573313483 [ 9046.016153] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9046.025782] LustreError: 230004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9046.638966] Lustre: server umount lustre-MDT0001 complete [ 9075.183324] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 9090.405659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9090.952797] LustreError: 234939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9090.988040] LustreError: 234939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9091.061727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9096.162448] LustreError: 234940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9096.185921] LustreError: 234940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9096.432353] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9101.289398] LustreError: 234939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9106.412248] LustreError: 234940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9107.865128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9108.120162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9114.075815] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9117.444980] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9125.393076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9125.737544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9132.817852] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9143.461061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9143.791921] Lustre: lustre-OST0001: new disk, initializing [ 9143.806813] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9143.818827] Lustre: 237246:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9144.023062] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9152.294646] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9152.612331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9152.644901] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9152.817371] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9165.650509] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9172.826036] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9190.148514] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 01:15:22 (1787807722) [ 9193.966225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.973512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9193.979368] Lustre: Skipped 2 previous similar messages [ 9193.985292] Lustre: Skipped 5 previous similar messages [ 9199.096623] LustreError: 234939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9199.130317] LustreError: 234939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9199.358309] Lustre: server umount lustre-MDT0000 complete [ 9204.839590] LustreError: 234919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787807738 with bad export cookie 13936684680573314365 [ 9204.858100] LustreError: 234919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9204.862984] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9205.293765] Lustre: server umount lustre-MDT0001 complete [ 9220.684987] Lustre: server umount lustre-OST0000 complete [ 9234.973627] Lustre: server umount lustre-OST0001 complete [ 9242.142486] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 9254.316187] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 9299.418956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.938387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9300.017439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9300.255331] Lustre: lustre-MDT0000: new disk, initializing [ 9300.398692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9300.464887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9306.742572] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9320.408127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9320.536542] Lustre: 241193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9320.585585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9320.590035] Lustre: Skipped 1 previous similar message [ 9320.658816] Lustre: lustre-MDT0001: new disk, initializing [ 9320.774801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9320.783064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9325.331645] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9335.973686] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9342.658723] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9350.987033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9351.433434] Lustre: lustre-OST0000: new disk, initializing [ 9351.443221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9351.456555] Lustre: 243164:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9352.947868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9352.954107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9353.136860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9359.449699] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9371.923549] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9378.444304] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9378.791101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9384.709616] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9384.916452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9389.030758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9389.031754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9389.032744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9389.067205] Lustre: Skipped 2 previous similar messages [ 9401.312135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9401.461572] Lustre: server umount lustre-OST0000 complete [ 9407.971042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9407.976690] LustreError: Skipped 1 previous similar message [ 9407.984915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9411.549295] Lustre: server umount lustre-MDT0000 complete [ 9414.630413] LustreError: 241205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9414.652489] LustreError: 241205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9415.873929] LustreError: 241184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787807949 with bad export cookie 13936684680573316507 [ 9415.883795] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9415.886609] LustreError: 241184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9416.156901] Lustre: server umount lustre-MDT0001 complete [ 9425.478299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9426.047302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9426.049842] Lustre: Skipped 2 previous similar messages [ 9431.405862] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9431.528717] LustreError: 245384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9431.557936] LustreError: 245384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9439.840771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9444.927896] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9453.526746] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9460.812781] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9468.303085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9476.371847] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9485.662644] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9633.762238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9633.790130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9633.813499] Lustre: Skipped 1 previous similar message [ 9633.820787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9633.826242] Lustre: Skipped 9 previous similar messages [ 9637.584211] Lustre: server umount lustre-OST0000 complete [ 9647.813215] Lustre: server umount lustre-MDT0000 complete [ 9649.124371] LustreError: 245385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9649.139693] LustreError: 245385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9651.885893] LustreError: 245364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787808185 with bad export cookie 13936684680573317382 [ 9651.889659] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9651.897824] LustreError: 245364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9652.508950] Lustre: server umount lustre-MDT0001 complete [ 9755.611258] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 9758.285584] Key type lgssc unregistered [ 9758.513737] LNet: 248996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9758.520867] LNetError: 248996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9759.593812] LNet: Removed LNI 192.168.201.119@tcp [ 9760.557413] Key type .llcrypt unregistered [ 9760.560061] Key type ._llcrypt unregistered [ 9787.609348] Key type ._llcrypt registered [ 9787.615189] Key type .llcrypt registered [ 9787.757495] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 9803.011935] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 9804.222778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9804.266294] alg: No test for adler32 (adler32-zlib) [ 9805.348416] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [ 9805.638229] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 9807.313469] Key type lgssc registered [ 9808.513132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9865.126560] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 9876.082086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9876.111548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9877.391345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9877.454523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9877.542768] Lustre: lustre-MDT0000: new disk, initializing [ 9877.616786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9877.633287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9882.037330] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9893.222372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9893.370553] Lustre: 253070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9893.477252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9893.482029] Lustre: Skipped 1 previous similar message [ 9893.574457] Lustre: lustre-MDT0001: new disk, initializing [ 9893.642388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9893.681873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9893.691185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9898.953986] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9904.272895] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9911.757981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9912.024757] Lustre: lustre-OST0000: new disk, initializing [ 9912.028650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9912.036270] Lustre: 254704:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9912.133661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9914.612291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9914.619933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9914.768407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9918.116674] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9929.230126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9929.403094] Lustre: lustre-OST0001: new disk, initializing [ 9929.410228] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9929.421217] Lustre: 255574:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9929.536841] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9936.282466] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 9936.589796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9936.597621] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9936.690914] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9948.416754] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9956.547857] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9970.658988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9970.667319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9970.697200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9972.707553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9972.709364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9976.242740] Lustre: server umount lustre-MDT0000 complete [ 9979.663422] LustreError: 256343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787808513 with bad export cookie 14132552903446564957 [ 9979.690360] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9980.003237] Lustre: server umount lustre-MDT0001 complete [ 9993.273099] Lustre: server umount lustre-OST0000 complete [10008.202273] Lustre: server umount lustre-OST0001 complete [10025.402456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10037.311874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10047.262707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10057.799859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10078.855363] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [10089.816769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10089.831997] Lustre: lustre-MDT0000: reset Object Index mappings [10090.321119] LustreError: 260586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10090.337517] LustreError: 260586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10090.377035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10094.858473] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10095.592448] LustreError: 260587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10100.708710] LustreError: 260586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10104.051805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10104.068905] Lustre: lustre-MDT0001: reset Object Index mappings [10104.469169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10109.454652] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10112.155479] Lustre: 261729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10119.191593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10127.335400] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10129.132788] LustreError: 262082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10134.526562] LustreError: 262083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10135.630355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10138.662354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10138.853763] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10138.859885] Lustre: Skipped 1 previous similar message [10144.246431] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10144.455333] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10152.890977] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10157.602344] Lustre: 263601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10164.710125] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10164.725809] Lustre: Skipped 4 previous similar messages [10164.727810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10164.765761] Lustre: Skipped 6 previous similar messages [10164.863135] Lustre: server umount lustre-MDT0000 complete [10169.066382] LustreError: 260566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787808703 with bad export cookie 14132552903446571747 [10169.071691] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10169.084024] LustreError: 260566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10169.609815] Lustre: server umount lustre-MDT0001 complete [10184.752344] Lustre: server umount lustre-OST0000 complete [10185.121111] Lustre: 250574:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787808703/real 1787808703] req@ffff9a6f011aa580 x1874652917003264/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787808719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10185.143857] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10185.158758] Lustre: Skipped 1 previous similar message [10190.265877] Lustre: server umount lustre-OST0001 complete [10215.250910] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:32:27 (1787808747) [10234.663303] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [10248.127161] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [10286.773653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10287.061867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10287.096741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10287.212722] Lustre: lustre-MDT0000: new disk, initializing [10287.285690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10287.312664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10292.032394] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10303.936122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10304.100052] Lustre: 267857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10304.179469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10304.187492] Lustre: Skipped 1 previous similar message [10304.339778] Lustre: lustre-MDT0001: new disk, initializing [10304.519195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10304.529531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10309.197850] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10318.371596] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10324.462689] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10331.868972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10332.156633] Lustre: lustre-OST0000: new disk, initializing [10332.163619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10332.170507] Lustre: 269828:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10334.139883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10334.157456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10334.300442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10339.756431] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10349.401265] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10353.936450] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10354.106470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10358.742336] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10358.937775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10364.912119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10364.913801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10364.916877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10364.925279] Lustre: Skipped 1 previous similar message [10364.940873] LustreError: Skipped 1 previous similar message [10370.023033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10370.037927] Lustre: Skipped 2 previous similar messages [10375.136534] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10375.329131] Lustre: server umount lustre-OST0000 complete [10380.270032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10380.279645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10380.290024] Lustre: Skipped 3 previous similar messages [10385.668480] Lustre: server umount lustre-MDT0000 complete [10389.222720] LustreError: 267851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787808923 with bad export cookie 14132552903446573840 [10389.227108] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10389.229561] LustreError: 267851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10389.638917] Lustre: server umount lustre-MDT0001 complete [10409.626777] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [10419.062083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10419.582811] LustreError: 272765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10419.607727] LustreError: 272765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10419.653947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10419.657138] Lustre: Skipped 2 previous similar messages [10424.824269] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10434.054198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10438.990209] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10441.769864] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10448.539429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10455.319967] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10464.726931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10464.969148] Lustre: lustre-OST0001: new disk, initializing [10464.976233] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10464.983619] Lustre: 275076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10466.710635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10466.722828] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10466.839601] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10470.992893] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10481.200920] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10485.766407] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10491.305647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10491.494711] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10501.093220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10501.110400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10501.135658] Lustre: Skipped 1 previous similar message [10501.146103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10501.156032] Lustre: Skipped 2 previous similar messages [10505.251798] Lustre: server umount lustre-MDT0000 complete [10507.757252] LustreError: 273494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10507.767107] LustreError: 273494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10509.544564] LustreError: 272747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787809043 with bad export cookie 14132552903446574715 [10509.560018] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10509.567025] LustreError: 272747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10509.933662] Lustre: server umount lustre-MDT0001 complete [10524.503186] Lustre: server umount lustre-OST0000 complete [10538.604299] Lustre: server umount lustre-OST0001 complete [10560.661975] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [10572.973671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10573.543239] LustreError: 278129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10573.615032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10573.630469] Lustre: Skipped 3 previous similar messages [10578.966052] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10588.461524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10594.615702] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10598.952474] Lustre: 279268:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10607.242670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10617.156518] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10618.173120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10627.833204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10633.740962] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10635.054441] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10643.229737] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10647.122897] Lustre: 281146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10652.265729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10652.388025] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10661.828758] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 01:39:54 (1787809194) [10679.784470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10679.785937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10679.801508] Lustre: Skipped 5 previous similar messages [10679.804566] Lustre: Skipped 4 previous similar messages [10683.363344] Lustre: server umount lustre-MDT0000 complete [10684.914595] LustreError: 278129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10684.932141] LustreError: 278129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10688.306038] LustreError: 278111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787809222 with bad export cookie 14132552903446581092 [10688.322064] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10688.328651] LustreError: 278111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10694.883097] Lustre: server umount lustre-MDT0001 complete [10713.057043] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10713.240863] Lustre: server umount lustre-OST0000 complete [10717.173451] Lustre: server umount lustre-OST0001 complete [10724.614916] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [10736.034452] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [10789.994917] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [10800.600899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.824199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10800.868658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10800.955815] Lustre: lustre-MDT0000: new disk, initializing [10801.106803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10806.310520] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10818.270522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.462142] Lustre: 285395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10818.841891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10824.326267] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10835.897680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10836.250949] Lustre: 287025:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10836.372691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10836.387147] Lustre: Skipped 5 previous similar messages [10838.411587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10842.223435] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10853.645931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10853.845993] Lustre: 287897:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10855.394272] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10855.414855] Lustre: Skipped 1 previous similar message [10855.585078] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10861.606187] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10874.278522] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10879.081513] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10879.087576] Lustre: Skipped 1 previous similar message [10894.624816] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 01:43:47 (1787809427) [10898.793915] Lustre: server umount lustre-MDT0000 complete [10900.965210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10900.974688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10900.986615] Lustre: Skipped 2 previous similar messages [10900.993096] LustreError: 285406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10901.017029] LustreError: 285406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10901.630401] LustreError: 287024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787809435 with bad export cookie 14132552903446588988 [10901.632354] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10901.641518] LustreError: 287024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10902.061334] Lustre: server umount lustre-MDT0001 complete [10915.896134] Lustre: server umount lustre-OST0000 complete [10929.652905] Lustre: server umount lustre-OST0001 complete [10938.071567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10942.446951] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10949.235427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10953.781924] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10961.401793] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10967.285918] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10973.939194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10980.412727] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [10989.035694] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11006.184410] loop1: detected capacity change from 0 to 204800000 [11006.291525] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11006.297587] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11006.304892] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11011.458786] loop1: detected capacity change from 0 to 204800000 [11017.677207] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11017.801567] LustreError: 290136:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs30145: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11017.829519] LustreError: 290136:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs30145 [11017.834386] LustreError: 290136:0:(mgs_handler.c:517:mgs_target_reg()) fs30145-MDT0001: Error checking target index: rc = -13 [11017.843634] LustreError: fs30145-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11017.863133] LustreError: 293867:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11017.870676] LustreError: 293867:0:(tgt_mount.c:2129:server_put_super()) no obd fs30145-MDT0001 [11017.875878] LustreError: 293867:0:(tgt_mount.c:116:server_deregister_mount()) fs30145-MDT0001 not registered [11017.907093] Lustre: server umount fs30145-MDT0001 complete [11017.909445] LustreError: 293867:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11026.256897] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11026.323516] Lustre: Setting parameter fs30145-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs30145-MDT0001 [11026.364694] Lustre: srv-fs30145-MDT0001: No data found on store. Initialize space. [11026.368570] Lustre: Skipped 4 previous similar messages [11026.428722] Lustre: fs30145-MDT0001: new disk, initializing [11026.431459] Lustre: Skipped 3 previous similar messages [11026.523604] Lustre: srv-fs30145-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11026.532978] Lustre: 294333:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30145-MDT0001-mdtlov: can't create new llog: rc = -115 [11026.543310] LustreError: 294333:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30145-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11031.106463] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11037.276326] Lustre: server umount fs30145-MDT0001 complete [11050.248684] loop1: detected capacity change from 0 to 204800000 [11055.294417] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11055.778390] Lustre: 295901:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30145-MDT0001-mdtlov: can't create new llog: rc = -115 [11055.786748] LustreError: 295901:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30145-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11060.209428] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11060.513059] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11065.649692] Lustre: server umount fs30145-MDT0001 complete [11075.558229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11075.563207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11075.576241] Lustre: Skipped 4 previous similar messages [11075.592321] Lustre: Skipped 9 previous similar messages [11089.376218] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11089.543078] Lustre: server umount lustre-MDT0000 complete [11099.199851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11099.432504] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11105.274260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11105.324450] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11115.089962] loop1: detected capacity change from 0 to 204800000 [11122.079700] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11122.470596] Lustre: 298346:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30145-MDT0001-mdtlov: can't create new llog: rc = -115 [11122.491323] LustreError: 298346:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30145-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11126.655732] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11131.469392] Lustre: server umount fs30145-MDT0001 complete [11158.498077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11158.507114] LustreError: Skipped 1 previous similar message [11161.576893] LustreError: 291974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.610180] LustreError: 291974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 116 previous similar messages [11164.095189] Lustre: server umount lustre-OST0000 complete [11176.361733] Lustre: server umount lustre-MDT0000 complete [11180.474583] LustreError: 290127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787809714 with bad export cookie 14132552903446591669 [11180.484054] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11180.496751] LustreError: 290127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11181.078934] Lustre: server umount lustre-MDT0001 complete [11195.881298] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [11199.113368] Key type lgssc unregistered [11199.410261] LNet: 300694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11199.431400] LNetError: 300694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11199.452161] LNet: Removed LNI 192.168.201.119@tcp [11200.660507] Key type .llcrypt unregistered [11200.662297] Key type ._llcrypt unregistered [11227.094165] Key type ._llcrypt registered [11227.096311] Key type .llcrypt registered [11227.313053] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [11228.701744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11228.851520] alg: No test for adler32 (adler32-zlib) [11230.048779] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [11230.306364] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [11232.032233] Key type lgssc registered [11233.293641] Lustre: Echo OBD driver; http://www.lustre.org/ [11241.343639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11241.369456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11242.794664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11248.170534] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11252.205519] LustreError: 301966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11252.250737] LustreError: 301966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11257.321042] LustreError: 301985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11259.009483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11259.490314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11263.785942] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11272.625337] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11279.524147] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11287.248246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11287.733029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11294.600979] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11302.861271] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11304.112675] LustreError: 303797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11309.548520] LustreError: 303798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11315.246750] LustreError: 303796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11315.267600] LustreError: 303796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11325.402522] LustreError: 303796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11325.430224] LustreError: 303796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11334.854967] loop1: detected capacity change from 0 to 204800000 [11335.165022] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11335.180416] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11335.227417] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11341.784275] loop1: detected capacity change from 0 to 204800000 [11343.338853] LustreError: 303798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11343.371121] LustreError: 303798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11348.020876] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11348.180775] Lustre: fs5561-OST0000: new disk, initializing [11348.185986] Lustre: srv-fs5561-OST0000: No data found on store. Initialize space. [11348.195483] Lustre: 305726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11348.310926] Lustre: fs5561-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11356.380713] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11368.139492] Lustre: server umount fs5561-OST0000 complete [11376.610749] LustreError: 303796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11376.650055] LustreError: 303796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11384.330976] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11385.242851] loop1: detected capacity change from 0 to 204800000 [11389.422625] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11393.611582] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11394.038035] Lustre: fs5561-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11401.747774] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11412.874332] loop2: detected capacity change from 0 to 204800000 [11413.209381] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11413.216681] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11413.224770] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11419.822548] loop2: detected capacity change from 0 to 204800000 [11427.329526] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11427.457416] LustreError: fs5561-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11427.474335] LustreError: fs5561-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11427.481668] LustreError: 308768:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11427.493744] LustreError: 308768:0:(tgt_mount.c:2129:server_put_super()) no obd fs5561-OST0001 [11427.508096] LustreError: 308768:0:(tgt_mount.c:116:server_deregister_mount()) fs5561-OST0001 not registered [11427.545312] Lustre: server umount fs5561-OST0001 complete [11427.547662] LustreError: 308768:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11438.428792] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11438.686966] Lustre: fs5561-OST0001: new disk, initializing [11438.693950] Lustre: srv-fs5561-OST0001: No data found on store. Initialize space. [11438.701539] Lustre: 309320:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11438.755934] Lustre: fs5561-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11440.618664] LustreError: 304469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11440.642814] LustreError: 304469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [11446.455279] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11456.150794] Lustre: server umount fs5561-OST0000 complete [11467.695814] Lustre: server umount fs5561-OST0001 complete [11485.105229] loop1: detected capacity change from 0 to 204800000 [11491.833105] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11492.040404] Lustre: fs5561-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11498.981204] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11506.692498] loop2: detected capacity change from 0 to 204800000 [11512.788880] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11519.396787] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11532.206481] Lustre: server umount fs5561-OST0000 complete [11537.945681] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11543.018414] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11543.600517] Lustre: server umount fs5561-OST0001 complete [11560.417956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11560.427368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11560.445470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11563.497236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11563.507633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11563.546544] Lustre: Skipped 1 previous similar message [11566.017595] Lustre: server umount lustre-OST0000 complete [11571.681232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11571.688632] LustreError: Skipped 1 previous similar message [11571.692864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11571.706057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11573.731445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11573.732994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11577.153475] Lustre: server umount lustre-MDT0000 complete [11578.857264] LustreError: 301962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11578.868804] LustreError: 301962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 72 previous similar messages [11581.720237] LustreError: 301946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787810115 with bad export cookie 9081691371464515423 [11581.735483] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11581.743797] LustreError: 301946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11582.245476] Lustre: server umount lustre-MDT0001 complete [11596.569951] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [11601.054925] Key type lgssc unregistered [11601.606514] LNet: 315249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11601.628307] LNetError: 315249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11601.672688] LNet: Removed LNI 192.168.201.119@tcp [11603.023172] Key type .llcrypt unregistered [11603.036920] Key type ._llcrypt unregistered [11638.200233] Key type ._llcrypt registered [11638.204242] Key type .llcrypt registered [11638.294268] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [11655.959216] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [11657.084488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11657.104699] alg: No test for adler32 (adler32-zlib) [11658.192927] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [11658.546195] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [11660.272204] Key type lgssc registered [11661.357717] Lustre: Echo OBD driver; http://www.lustre.org/ [11703.002188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11703.035727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11704.293524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11704.332628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11704.418122] Lustre: lustre-MDT0000: new disk, initializing [11704.517173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11704.619676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11709.720553] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11722.013311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11722.148968] Lustre: 319112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11722.225147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11722.230415] Lustre: Skipped 1 previous similar message [11722.315396] Lustre: lustre-MDT0001: new disk, initializing [11722.445488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11722.512731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11722.532740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11728.449548] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11738.529089] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11745.116453] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11752.012529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11752.351658] Lustre: lustre-OST0000: new disk, initializing [11752.357832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11752.366588] Lustre: 321081:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11752.445866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11758.664382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11758.682149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11758.832583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11759.254259] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [11769.715317] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11772.021450] Lustre: Mounted lustre-client - version 2.17.57_53_g458ba6d [11773.631261] Lustre: Unmounted lustre-client [11787.244667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11787.250303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11787.267643] Lustre: Skipped 1 previous similar message [11787.299340] Lustre: Skipped 1 previous similar message [11791.303208] Lustre: server umount lustre-OST0000 complete [11797.474424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11797.483896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11797.506989] Lustre: Skipped 1 previous similar message [11797.513821] Lustre: Skipped 1 previous similar message [11802.599133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11803.148923] Lustre: server umount lustre-MDT0000 complete [11807.586238] LustreError: 321080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787810341 with bad export cookie 12203128740934402752 [11807.597300] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11807.597937] LustreError: 321080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11807.974274] Lustre: server umount lustre-MDT0001 complete [11823.783663] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [11827.556217] LNet: 1 local NIs in recovery (showing 1): 192.168.201.119@tcp [11828.387527] Key type lgssc unregistered [11828.847833] LNet: 323242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11828.862432] LNetError: 323242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11904.684724] LNet: Removed LNI 192.168.201.119@tcp [11905.677475] Key type .llcrypt unregistered [11905.683294] Key type ._llcrypt unregistered [11925.322465] Key type ._llcrypt registered [11925.332612] Key type .llcrypt registered [11925.463068] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [11942.489828] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [11943.832618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11943.929092] alg: No test for adler32 (adler32-zlib) [11944.916211] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [11945.175305] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [11946.880197] Key type lgssc registered [11948.023089] Lustre: Echo OBD driver; http://www.lustre.org/ [11978.336457] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:01:51 (1787810511) [11984.444940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11984.467677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11985.875648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11985.906967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11986.024566] Lustre: lustre-MDT0000: new disk, initializing [11986.179636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11986.202629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11990.367981] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12001.246599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12001.381932] Lustre: 326479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12001.434520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12001.438596] Lustre: Skipped 1 previous similar message [12001.524640] Lustre: lustre-MDT0001: new disk, initializing [12001.634166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12001.665173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12001.682851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12006.241039] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12015.306519] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12022.028876] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12030.125089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12030.517384] Lustre: lustre-OST0000: new disk, initializing [12030.522973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12030.540127] Lustre: 328444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12030.717769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12035.852278] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12037.170803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12037.179834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12037.317959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12044.408807] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12063.205175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12063.208320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12063.223811] LustreError: Skipped 1 previous similar message [12063.224851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12063.244665] Lustre: Skipped 1 previous similar message [12066.458981] Lustre: server umount lustre-OST0000 complete [12073.442646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12073.446465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12073.462014] Lustre: Skipped 1 previous similar message [12073.465354] Lustre: Skipped 1 previous similar message [12073.490279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12076.391092] Lustre: server umount lustre-MDT0000 complete [12080.855355] LustreError: 326471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787810614 with bad export cookie 12798566468737376792 [12080.875161] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12081.377807] Lustre: server umount lustre-MDT0001 complete [12093.868223] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [12097.012820] Key type lgssc unregistered [12097.334722] LNet: 330357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12097.343391] LNetError: 330357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12097.363334] LNet: Removed LNI 192.168.201.119@tcp [12098.399190] Key type .llcrypt unregistered [12098.403488] Key type ._llcrypt unregistered [12119.272828] Key type ._llcrypt registered [12119.274745] Key type .llcrypt registered [12119.372609] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [12120.047521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12120.117241] alg: No test for adler32 (adler32-zlib) [12121.262501] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [12121.468462] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [12123.120165] Key type lgssc registered [12124.099933] Lustre: Echo OBD driver; http://www.lustre.org/ [12132.009122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12132.041290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12133.613544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12137.840564] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12143.082277] LustreError: 331631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12143.108031] LustreError: 331631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12146.818417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12147.115954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12151.195321] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12159.539413] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12165.014654] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12171.849793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12172.110123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12179.195858] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12186.816274] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12197.825866] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12199.729189] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12204.009645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12204.013557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12204.026140] Lustre: Skipped 1 previous similar message [12204.038662] Lustre: Skipped 1 previous similar message [12205.063989] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping) [12208.610169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12208.620604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12208.635481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12210.067227] Lustre: server umount lustre-MDT0000 complete [12210.137731] LustreError: 334168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12214.242631] LustreError: 331972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12214.278497] LustreError: 331972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12214.732788] LustreError: 331612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787810748 with bad export cookie 17128827400990684709 [12214.769157] LustreError: 331612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12214.771479] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12215.295861] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.19@tcp (stopping) [12215.300359] Lustre: Skipped 3 previous similar messages [12219.390080] LustreError: 331626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12219.402510] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12219.437964] LustreError: 331626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12219.478237] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12221.728279] Lustre: server umount lustre-MDT0001 complete [12227.180608] Lustre: server umount lustre-OST0000 complete [12238.982801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12245.096435] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12247.934452] Lustre: 335811:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.119@tcp [12250.058484] Lustre: server umount MGS complete [12263.057498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12263.821252] LustreError: 336384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12263.848805] LustreError: 336384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12263.956608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12270.558963] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12273.140410] LustreError: 336385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12273.160905] LustreError: 336385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12280.505176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12280.842018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12285.432845] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12294.037379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12294.324857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12300.147527] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12308.543332] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12310.498239] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12316.861893] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12318.746475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12324.328316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12324.330453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12324.337071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12324.345844] Lustre: Skipped 2 previous similar messages [12326.057534] Lustre: server umount lustre-OST0000 complete [12332.003512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12332.011124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12332.025144] Lustre: Skipped 1 previous similar message [12335.977540] Lustre: server umount lustre-MDT0000 complete [12339.683266] LustreError: 336380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12339.707562] LustreError: 336380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12340.292531] LustreError: 336365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787810874 with bad export cookie 17128827400990685542 [12340.323297] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12341.032308] Lustre: server umount lustre-MDT0001 complete [12354.571439] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [12357.381689] Key type lgssc unregistered [12357.647954] LNet: 339824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12357.661919] LNetError: 339824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12357.688852] LNet: Removed LNI 192.168.201.119@tcp [12358.656922] Key type .llcrypt unregistered [12358.658511] Key type ._llcrypt unregistered [12382.039291] Key type ._llcrypt registered [12382.041397] Key type .llcrypt registered [12382.246826] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [12398.155190] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [12399.539641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12399.601959] alg: No test for adler32 (adler32-zlib) [12400.644975] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [12400.858580] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [12402.544176] Key type lgssc registered [12403.847918] Lustre: Echo OBD driver; http://www.lustre.org/ [12446.509773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12446.546478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12447.825307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12447.848538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12447.894687] Lustre: lustre-MDT0000: new disk, initializing [12447.943778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12447.955776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12453.032775] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12464.083294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12464.220940] Lustre: 342991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12464.306766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12464.310472] Lustre: Skipped 1 previous similar message [12464.383741] Lustre: lustre-MDT0001: new disk, initializing [12464.450623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12464.490493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12464.502911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12469.542456] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12478.766141] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12485.432577] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12492.556491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12492.866862] Lustre: lustre-OST0000: new disk, initializing [12492.877051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12492.884953] Lustre: 344959:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12492.990104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12499.247172] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12502.083450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12502.096542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12502.228568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12509.029607] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12515.063100] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12515.378845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12521.181876] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12521.372323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12528.106235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12528.109493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12528.152393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12528.173697] LustreError: Skipped 1 previous similar message [12533.222931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12533.240353] Lustre: Skipped 2 previous similar messages [12538.336146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12538.344535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12538.374643] Lustre: Skipped 1 previous similar message [12538.834899] Lustre: server umount lustre-OST0000 complete [12546.529706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12546.534728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12546.545706] Lustre: Skipped 1 previous similar message [12546.548413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12548.584427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12553.699691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12553.707783] Lustre: Skipped 3 previous similar messages [12558.304189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12558.953645] Lustre: server umount lustre-MDT0000 complete [12563.204137] LustreError: 342983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787811097 with bad export cookie 14446697794943936982 [12563.205513] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12563.214233] LustreError: 342983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12563.688165] Lustre: server umount lustre-MDT0001 complete [12574.855841] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:11:47 (1787811107) [12594.676832] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [12606.981770] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [12644.967373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12645.304976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12645.332427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12645.382435] Lustre: lustre-MDT0000: new disk, initializing [12645.453937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12645.473842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12651.118594] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12660.792274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12660.915388] Lustre: 349722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12660.973607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12660.987605] Lustre: Skipped 1 previous similar message [12661.077789] Lustre: lustre-MDT0001: new disk, initializing [12661.286868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12661.342615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12661.359089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12665.609511] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12676.532781] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12682.037685] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12689.317539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12689.638679] Lustre: lustre-OST0000: new disk, initializing [12689.642105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12689.647591] Lustre: 351689:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12689.739398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12696.002063] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [12698.696133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12698.705932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12698.808341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12706.026146] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12745.187470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12745.221366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12745.231859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12745.256656] Lustre: Skipped 1 previous similar message [12747.226813] Lustre: server umount lustre-OST0000 complete [12753.378779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12753.386427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12753.426037] Lustre: Skipped 1 previous similar message [12758.651374] Lustre: server umount lustre-MDT0000 complete [12760.562325] LustreError: 349734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12760.589677] LustreError: 349734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12763.347369] LustreError: 349714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787811297 with bad export cookie 14446697794943937864 [12763.366222] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12763.920731] Lustre: server umount lustre-MDT0001 complete [12777.873443] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [12781.154308] Key type lgssc unregistered [12781.512735] LNet: 353865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12781.536968] LNetError: 353865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12781.557186] LNet: Removed LNI 192.168.201.119@tcp [12782.532915] Key type .llcrypt unregistered [12782.535039] Key type ._llcrypt unregistered [12881.137864] Key type ._llcrypt registered [12881.140290] Key type .llcrypt registered [12881.324674] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [12881.806435] Key type .llcrypt unregistered [12881.809443] Key type ._llcrypt unregistered [12966.591389] Key type ._llcrypt registered [12966.592540] Key type .llcrypt registered [12966.681706] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [12966.909450] Key type .llcrypt unregistered [12966.910550] Key type ._llcrypt unregistered [12997.108189] Key type ._llcrypt registered [12997.110252] Key type .llcrypt registered [12997.261618] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [12997.530635] Key type .llcrypt unregistered [12997.532797] Key type ._llcrypt unregistered [13030.154876] Key type ._llcrypt registered [13030.157631] Key type .llcrypt registered [13030.307271] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [13030.867314] Key type .llcrypt unregistered [13030.869108] Key type ._llcrypt unregistered [13115.842754] Key type ._llcrypt registered [13115.845836] Key type .llcrypt registered [13115.981775] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [13116.297386] Key type .llcrypt unregistered [13116.301312] Key type ._llcrypt unregistered [13147.992312] Key type ._llcrypt registered [13147.995809] Key type .llcrypt registered [13148.167418] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [13148.557957] Key type .llcrypt unregistered [13148.565084] Key type ._llcrypt unregistered [13175.611564] Key type ._llcrypt registered [13175.613265] Key type .llcrypt registered [13175.690698] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [13175.867292] Key type .llcrypt unregistered [13175.869041] Key type ._llcrypt unregistered [13201.871729] Key type ._llcrypt registered [13201.872705] Key type .llcrypt registered [13201.957858] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [13202.230308] Key type .llcrypt unregistered [13202.231666] Key type ._llcrypt unregistered [13237.499879] Key type ._llcrypt registered [13237.503219] Key type .llcrypt registered [13237.608611] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [13237.935388] Key type .llcrypt unregistered [13237.938062] Key type ._llcrypt unregistered [13269.758882] Key type ._llcrypt registered [13269.760928] Key type .llcrypt registered [13269.859335] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [13270.246400] Key type .llcrypt unregistered [13270.248115] Key type ._llcrypt unregistered [13327.802547] Key type ._llcrypt registered [13327.804743] Key type .llcrypt registered [13327.883932] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [13328.122289] Key type .llcrypt unregistered [13328.124342] Key type ._llcrypt unregistered [13359.198712] Key type ._llcrypt registered [13359.203390] Key type .llcrypt registered [13359.413710] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [13359.848389] Key type .llcrypt unregistered [13359.850917] Key type ._llcrypt unregistered [13396.049826] Key type ._llcrypt registered [13396.052421] Key type .llcrypt registered [13396.152385] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [13396.558434] Key type .llcrypt unregistered [13396.563859] Key type ._llcrypt unregistered [13457.113202] Key type ._llcrypt registered [13457.115354] Key type .llcrypt registered [13457.244846] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [13472.209382] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [13473.291178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13473.328590] alg: No test for adler32 (adler32-zlib) [13474.427226] Lustre: Lustre: Build Version: 2.17.57_53_g458ba6d [13474.642495] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [13476.296195] Key type lgssc registered [13477.529921] Lustre: Echo OBD driver; http://www.lustre.org/ [13512.196510] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13512.231542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13513.533117] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13513.649298] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13513.832874] Lustre: tst250fs-MDT0000: new disk, initializing [13513.891188] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13513.901807] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13518.584598] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [13529.678525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13529.817830] Lustre: 374254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13529.912367] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13529.918719] Lustre: Skipped 1 previous similar message [13530.011236] Lustre: tst250fs-MDT0001: new disk, initializing [13530.155613] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13530.208963] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13530.225709] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13535.294040] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [13540.593947] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13547.971614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13548.268683] Lustre: tst250fs-OST0000: new disk, initializing [13548.272491] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13548.286406] Lustre: 375885:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13548.419707] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13553.267916] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13553.283646] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13553.408044] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13554.770675] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [13565.632042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13565.813878] Lustre: tst250fs-OST0001: new disk, initializing [13565.817213] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13565.822963] Lustre: 376757:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13565.934133] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13572.224617] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [13573.731847] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13573.746588] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13573.882814] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13578.298774] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13581.435769] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13581.439355] Lustre: Skipped 2 previous similar messages [13599.726395] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13599.734251] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13599.748217] Lustre: Skipped 1 previous similar message [13604.399773] Lustre: server umount tst250fs-MDT0000 complete [13607.549333] LustreError: 375884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787812141 with bad export cookie 5281802206970692463 [13607.552599] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13607.554133] LustreError: 375884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13607.867503] Lustre: server umount tst250fs-MDT0001 complete [13622.597480] Lustre: server umount tst250fs-OST0000 complete [13635.128591] Lustre: server umount tst250fs-OST0001 complete [13652.147176] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [13662.360540] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [13693.129959] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:30:25 (1787812225) [13694.635172] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13711.413262] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [13721.551341] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [13757.368848] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13522 sec ========= 02:31:29 (1787812289) [13759.923058] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:31:32 (1787812292) === [13764.859648] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:31:37 (1787812297) === [13811.708212] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [13815.007326] Key type lgssc unregistered [13815.274802] LNet: 385911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13815.285579] LNetError: 385911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13816.359686] LNet: Removed LNI 192.168.201.119@tcp [13817.417154] Key type .llcrypt unregistered [13817.423176] Key type ._llcrypt unregistered