[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 620459664 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002647] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005086] kvm-guest: setup PV IPIs [ 0.009772] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011023] pid_max: default: 32768 minimum: 301 [ 0.013045] LSM: Security Framework initializing [ 0.015073] Yama: becoming mindful. [ 0.016064] SELinux: Initializing. [ 0.017111] *** VALIDATE selinux *** [ 0.027211] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.033506] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.034251] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036198] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037187] *** VALIDATE tmpfs *** [ 0.039178] *** VALIDATE proc *** [ 0.041331] *** VALIDATE cgroup *** [ 0.042020] *** VALIDATE cgroup2 *** [ 0.044416] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.046267] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.047090] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.048054] Spectre V2 : User space: Vulnerable [ 0.049015] Speculative Store Bypass: Vulnerable [ 0.052745] debug: unmapping init [mem 0xffffffff86e59000-0xffffffff86e60fff] [ 0.053902] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.055887] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.056036] ... version: 2 [ 0.057030] ... bit width: 48 [ 0.058083] ... generic registers: 4 [ 0.059025] ... value mask: 0000ffffffffffff [ 0.060027] ... max period: 00007fffffffffff [ 0.061028] ... fixed-purpose events: 3 [ 0.062025] ... event mask: 000000070000000f [ 0.063424] rcu: Hierarchical SRCU implementation. [ 0.066137] smp: Bringing up secondary CPUs ... [ 0.067818] x86: Booting SMP configuration: [ 0.068051] .... node #0, CPUs: #1 #2 #3 [ 0.075181] smp: Brought up 1 node, 4 CPUs [ 0.077037] smpboot: Max logical packages: 1 [ 0.078032] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149154] node 0 deferred pages initialised in 69ms [ 0.152191] devtmpfs: initialized [ 0.153400] x86/mm: Memory block size: 128MB [ 0.156170] gcov: version magic: 0x41383552 [ 0.158456] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159124] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160409] pinctrl core: initialized pinctrl subsystem [ 0.161303] [ 0.162000] ************************************************************* [ 0.163021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164019] ** ** [ 0.165018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166020] ** ** [ 0.167024] ** This means that this kernel is built to expose internal ** [ 0.168016] ** IOMMU data structures, which may compromise security on ** [ 0.169029] ** your system. ** [ 0.170032] ** ** [ 0.171022] ** If you see this message and you are not debugging the ** [ 0.172016] ** kernel, report this immediately to your vendor! ** [ 0.173021] ** ** [ 0.174023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175021] ************************************************************* [ 0.177201] NET: Registered protocol family 16 [ 0.178679] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179098] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182052] cpuidle: using governor menu [ 0.184895] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188892] PCI: Using configuration type 1 for base access [ 0.192228] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.204425] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206146] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.211095] cryptd: max_cpu_qlen set to 1000 [ 0.213233] ACPI: Added _OSI(Module Device) [ 0.214017] ACPI: Added _OSI(Processor Device) [ 0.215021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.216023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.220975] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224487] ACPI: Interpreter enabled [ 0.225098] ACPI: PM: (supports S0 S3 S4 S5) [ 0.226019] ACPI: Using IOAPIC for interrupt routing [ 0.227159] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.228522] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.243064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.245188] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.252903] acpiphp: Slot [2] registered [ 0.255217] acpiphp: Slot [5] registered [ 0.257219] acpiphp: Slot [6] registered [ 0.259471] acpiphp: Slot [7] registered [ 0.262234] acpiphp: Slot [8] registered [ 0.264212] acpiphp: Slot [9] registered [ 0.266210] acpiphp: Slot [10] registered [ 0.268364] acpiphp: Slot [3] registered [ 0.271166] acpiphp: Slot [4] registered [ 0.273270] acpiphp: Slot [11] registered [ 0.275145] acpiphp: Slot [12] registered [ 0.277202] acpiphp: Slot [13] registered [ 0.279403] acpiphp: Slot [14] registered [ 0.282165] acpiphp: Slot [15] registered [ 0.284155] acpiphp: Slot [16] registered [ 0.286182] acpiphp: Slot [17] registered [ 0.288144] acpiphp: Slot [18] registered [ 0.290120] acpiphp: Slot [19] registered [ 0.292335] acpiphp: Slot [20] registered [ 0.294232] acpiphp: Slot [21] registered [ 0.296171] acpiphp: Slot [22] registered [ 0.299187] acpiphp: Slot [23] registered [ 0.301180] acpiphp: Slot [24] registered [ 0.303192] acpiphp: Slot [25] registered [ 0.305188] acpiphp: Slot [26] registered [ 0.307142] acpiphp: Slot [27] registered [ 0.309161] acpiphp: Slot [28] registered [ 0.311148] acpiphp: Slot [29] registered [ 0.312112] acpiphp: Slot [30] registered [ 0.314143] acpiphp: Slot [31] registered [ 0.316130] PCI host bridge to bus 0000:00 [ 0.319035] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322042] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324136] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.332036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338226] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342556] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.346982] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.359987] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.366010] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.372029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.379993] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383015] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.386059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.388343] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.415028] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.424075] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.435257] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.446024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.457028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.507029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.542118] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.564022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.579024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.609021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.626523] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.634022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.643026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.673000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.700057] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.711022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.718000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.725161] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.740436] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.755227] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.784029] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.816021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.834651] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.846025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.862040] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.887024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.901740] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.906652] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.910670] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.913586] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.917364] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.926155] iommu: Default domain type: Passthrough [ 0.932214] SCSI subsystem initialized [ 0.935243] ACPI: bus type USB registered [ 0.937480] usbcore: registered new interface driver usbfs [ 0.940127] usbcore: registered new interface driver hub [ 0.944251] usbcore: registered new device driver usb [ 0.952649] pps_core: LinuxPPS API ver. 1 registered [ 0.955019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.966085] PTP clock support registered [ 0.967464] EDAC MC: Ver: 3.0.0 [ 0.969000] PCI: Using ACPI for IRQ routing [ 0.971000] NetLabel: Initializing [ 0.973017] NetLabel: domain hash size = 128 [ 0.978018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.991401] NetLabel: unlabeled traffic allowed by default [ 1.005188] vgaarb: loaded [ 1.009875] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.017020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.029246] clocksource: Switched to clocksource kvm-clock [ 1.344874] VFS: Disk quotas dquot_6.6.0 [ 1.347725] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.351677] *** VALIDATE ramfs *** [ 1.353530] *** VALIDATE hugetlbfs *** [ 1.355732] pnp: PnP ACPI init [ 1.359444] pnp: PnP ACPI: found 6 devices [ 1.383407] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.388660] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.392234] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.395298] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.399452] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.402366] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.406796] NET: Registered protocol family 2 [ 1.410541] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.418980] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.424469] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.432311] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.436996] TCP: Hash tables configured (established 65536 bind 65536) [ 1.440656] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.445354] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.449825] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.455655] NET: Registered protocol family 1 [ 1.460519] RPC: Registered named UNIX socket transport module. [ 1.465476] RPC: Registered udp transport module. [ 1.469987] RPC: Registered tcp transport module. [ 1.472580] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.477853] NET: Registered protocol family 44 [ 1.482546] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.486443] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.523639] pci 0000:00:00.0: quirk_passive_release+0x0/0x90 took 36374 usecs [ 1.529670] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.537371] PCI: CLS 0 bytes, default 64 [ 1.545284] Unpacking initramfs... [ 3.818481] debug: unmapping init [mem 0xffff8b43fcc54000-0xffff8b43fffbffff] [ 3.824922] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.829088] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.835428] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.317165] Initialise system trusted keyrings [ 5.329652] Key type blacklist registered [ 5.358466] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.579582] zbud: loaded [ 5.595872] *** VALIDATE nfs *** [ 5.601184] *** VALIDATE nfs4 *** [ 5.609995] pstore: using deflate compression [ 5.623710] Platform Keyring initialized [ 5.834453] NET: Registered protocol family 38 [ 5.836495] Key type asymmetric registered [ 5.840461] Asymmetric key parser 'x509' registered [ 5.842654] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.845846] io scheduler mq-deadline registered [ 5.847522] io scheduler kyber registered [ 5.849409] io scheduler bfq registered [ 5.851456] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.855176] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.858985] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.862875] ACPI: Power Button [PWRF] [ 5.870253] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.878291] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.909180] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.915973] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.933680] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.963306] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.994340] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.002496] Non-volatile memory driver v1.3 [ 6.004582] Linux agpgart interface v0.103 [ 6.037668] virtio_blk virtio1: [vda] 149376 512-byte logical blocks (76.5 MB/72.9 MiB) [ 6.040582] vda: detected capacity change from 0 to 76480512 [ 6.054874] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.058346] vdb: detected capacity change from 0 to 1073741824 [ 6.076023] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.079910] vdc: detected capacity change from 0 to 2621440000 [ 6.105399] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.109065] vdd: detected capacity change from 0 to 2621440000 [ 6.134079] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.137940] vde: detected capacity change from 0 to 4294967296 [ 6.160669] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.164987] vdf: detected capacity change from 0 to 4294967296 [ 6.176303] libphy: Fixed MDIO Bus: probed [ 6.187364] usbcore: registered new interface driver usbserial_generic [ 6.192402] usbserial: USB Serial support registered for generic [ 6.197868] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.203406] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.205862] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.208975] mousedev: PS/2 mouse device common for all mice [ 6.213949] rtc_cmos 00:05: RTC can wake from S4 [ 6.219473] rtc_cmos 00:05: registered as rtc0 [ 6.221776] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.223802] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.231964] intel_pstate: CPU model not supported [ 6.235381] hid: raw HID events driver (C) Jiri Kosina [ 6.248598] usbcore: registered new interface driver usbhid [ 6.250706] usbhid: USB HID core driver [ 6.253300] drop_monitor: Initializing network drop monitor service [ 6.256193] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.258013] Initializing XFRM netlink socket [ 6.259490] NET: Registered protocol family 10 [ 6.269901] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.278083] Segment Routing with IPv6 [ 6.279818] NET: Registered protocol family 17 [ 6.282108] mpls_gso: MPLS GSO support [ 6.287590] RAS: Correctable Errors collector initialized. [ 6.290122] AVX version of gcm_enc/dec engaged. [ 6.292104] AES CTR mode by8 optimization enabled [ 6.443367] sched_clock: Marking stable (6443290935, 0)->(8079634235, -1636343300) [ 6.451499] registered taskstats version 1 [ 6.454993] Loading compiled-in X.509 certificates [ 6.458497] zswap: loaded using pool lzo/zbud [ 6.497949] Key type big_key registered [ 6.519198] Key type encrypted registered [ 6.523239] ima: No TPM chip found, activating TPM-bypass! [ 6.528268] ima: Allocated hash algorithm: sha1 [ 6.531326] ima: No architecture policies found [ 6.534255] evm: Initialising EVM extended attributes: [ 6.536635] evm: security.selinux [ 6.538516] evm: security.ima [ 6.539882] evm: security.capability [ 6.541822] evm: HMAC attrs: 0x1 [ 6.545728] rtc_cmos 00:05: setting system clock to 2026-08-11 02:04:16 UTC (1786413856) [ 6.558485] debug: unmapping init [mem 0xffffffff87e03000-0xffffffff87ffffff] [ 6.563281] debug: unmapping init [mem 0xffffffff86b82000-0xffffffff86e58fff] [ 6.576232] Write protecting the kernel read-only data: 28672k [ 6.580282] debug: unmapping init [mem 0xffffffff85203000-0xffffffff853fffff] [ 6.583704] debug: unmapping init [mem 0xffffffff85b14000-0xffffffff85bfffff] [ 6.647803] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.665866] systemd[1]: Detected virtualization kvm. [ 6.670498] systemd[1]: Detected architecture x86-64. [ 6.672952] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.706248] systemd[1]: No hostname configured. [ 6.708830] systemd[1]: Set hostname to . [ 6.711578] random: systemd: uninitialized urandom read (16 bytes read) [ 6.715495] systemd[1]: Initializing machine ID from random generator. [ 6.834345] random: ln: uninitialized urandom read (6 bytes read) [ 7.017360] random: systemd: uninitialized urandom read (16 bytes read) [ 7.020762] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 7.027738] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 7.036649] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.796223] device-mapper: uevent: version 1.0.3 [ 8.798958] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ 10.573293] virtio_net virtio0 ens2: renamed from eth0 [[ 10.763563] random: fast init done  OK ] Mounted Kernel Configuration File System. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 10.854489] scsi host0: ata_piix [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.910093] scsi host1: ata_piix [ 10.921266] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.939808] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.131689] random: crng init done [ 16.133960] random: 7 urandom warning(s) missed due to ratelimiting [ 18.215063] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.750228] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.158607] printk: systemd: 23 output lines suppressed due to ratelimiting [ 22.768952] SELinux: Disabled at runtime. [ 22.863514] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.883914] systemd[1]: Detected virtualization kvm. [ 22.887299] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.457370] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.468886] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.488727] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.502723] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.512933] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.527391] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.547077] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables.[ 25.038692] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS .. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 26.116254] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.359482] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.003329] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.041215] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.820555] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 34.063133] Key type dns_resolver registered [ 34.658910] NFS: Registering the id_resolver key type [ 34.664829] Key type id_resolver registered [ 34.669909] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg317-server login: [ 92.834349] hrtimer: interrupt took 3021883 ns [ 97.313962] libcfs: loading out-of-tree module taints kernel. [ 97.364483] Key type ._llcrypt registered [ 97.370460] Key type .llcrypt registered [ 97.485469] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 115.390277] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 117.297603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.315298] alg: No test for adler32 (adler32-zlib) [ 119.003237] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 119.715642] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 121.559294] Key type lgssc registered [ 123.070980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 137.657362] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 176.812711] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 188.619265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 188.645279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.838699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.866720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 189.934822] Lustre: lustre-MDT0000: new disk, initializing [ 189.997184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.017275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.950853] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.351474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.486191] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 212.536856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 212.542140] Lustre: Skipped 1 previous similar message [ 212.636682] Lustre: lustre-MDT0001: new disk, initializing [ 212.724949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 212.748310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 212.763600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 217.269735] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.661311] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 232.948928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.244529] Lustre: lustre-OST0000: new disk, initializing [ 233.250380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 233.272594] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.389949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 239.204661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 239.222845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 239.412617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.510563] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.672906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.800158] Lustre: lustre-OST0001: new disk, initializing [ 256.803499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 256.810239] Lustre: 9522:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.867588] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 262.646513] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.271420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 263.280216] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 263.514512] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 274.733863] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 285.231990] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 292.734926] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing check_logdir /tmp/testlogs/ [ 298.501963] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing yml_node [ 303.447496] Lustre: DEBUG MARKER: Client: 2.17.54.107 [ 307.118745] Lustre: DEBUG MARKER: MDS: 2.17.54.107 [ 311.503900] Lustre: DEBUG MARKER: OSS: 2.17.54.107 [ 313.487266] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 10 22:09:21 EDT 2026 [ 330.664963] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 332.363649] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 350.179833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.184873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.216356] Lustre: Skipped 1 previous similar message [ 350.241858] Lustre: Skipped 3 previous similar messages [ 354.042468] Lustre: server umount lustre-MDT0000 complete [ 358.593064] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414208 with bad export cookie 54183537859213651 [ 358.601201] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.607031] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 360.429546] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.429740] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.443748] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 360.454284] Lustre: Skipped 3 previous similar messages [ 360.465519] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 360.471941] Lustre: Skipped 1 previous similar message [ 363.489975] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.490765] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 363.510806] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 363.527239] Lustre: Skipped 1 previous similar message [ 365.023612] Lustre: server umount lustre-MDT0001 complete [ 370.075501] Lustre: server umount lustre-OST0000 complete [ 375.893804] Lustre: server umount lustre-OST0001 complete [ 383.565294] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 393.484058] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 432.862333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.240398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 433.272043] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 433.441575] Lustre: lustre-MDT0000: new disk, initializing [ 433.658605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 433.708758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 438.476582] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 449.422280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.590374] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 449.689625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 449.695341] Lustre: Skipped 1 previous similar message [ 449.785683] Lustre: lustre-MDT0001: new disk, initializing [ 449.887936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 449.952875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 449.981715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 454.324195] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 464.654918] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 471.289973] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 478.560181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 478.797542] Lustre: lustre-OST0000: new disk, initializing [ 478.801834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 478.808574] Lustre: 17047:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 478.898475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 480.022187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 480.039888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 480.179495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 485.884653] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 496.074702] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 500.899693] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 501.273175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 506.008472] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 506.356499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 510.944519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.951656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.968953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.969265] Lustre: Skipped 1 previous similar message [ 511.063487] Lustre: Skipped 1 previous similar message [ 514.712295] Lustre: server umount lustre-OST0000 complete [ 521.246547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.253866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 521.260770] Lustre: Skipped 1 previous similar message [ 525.931149] Lustre: server umount lustre-MDT0000 complete [ 526.311287] LustreError: 15085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.321289] LustreError: 15085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 529.955887] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414379 with bad export cookie 54183537859215184 [ 529.971377] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.976203] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 530.038144] LustreError: 3645:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 530.394851] Lustre: server umount lustre-MDT0001 complete [ 537.157801] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:13:05 (1786414385) [ 558.643885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.001642] LustreError: 19921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.058219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.230103] LustreError: 19922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.274225] LustreError: 19922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 564.861717] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 573.409205] LustreError: 19921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.436536] LustreError: 19921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 575.538737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.894802] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 587.704411] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 592.832502] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 594.660339] Lustre: Failing over lustre-MDT0000 [ 594.887429] Lustre: server umount lustre-MDT0000 complete [ 595.429082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.439818] LustreError: 19916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.456757] Lustre: Skipped 2 previous similar messages [ 595.480582] LustreError: 19916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 611.820696] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786414445/real 1786414445] req@ffff8b447ed2ce00 x1873190814125056/t0(0) o400->MGC192.168.203.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786414461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 611.852263] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.847544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.089582] LustreError: 3644:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b447ed2fb80 x1873190814127744/t0(0) o250->MGC192.168.203.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 621.638465] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 625.382735] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 626.660859] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 626.668181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 626.704796] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 636.368150] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 642.016934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 642.018244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 642.019981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 642.034597] Lustre: Skipped 1 previous similar message [ 642.037606] LustreError: Skipped 1 previous similar message [ 654.303546] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 654.420604] Lustre: server umount lustre-MDT0000 complete [ 656.370136] LustreError: 20650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 656.378564] LustreError: 20650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 658.557941] LustreError: 19904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414508 with bad export cookie 54183537859216598 [ 658.566632] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 658.858581] Lustre: server umount lustre-MDT0001 complete [ 667.807661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.167391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.172212] Lustre: Skipped 2 previous similar messages [ 673.504744] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 682.373613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.871404] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 694.842698] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 700.355419] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 702.924717] Lustre: Failing over lustre-MDT0000 [ 703.186678] Lustre: server umount lustre-MDT0000 complete [ 703.456803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 703.466161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.338284] Lustre: 3648:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786414553/real 1786414553] req@ffff8b44803caa00 x1873190814154624/t0(0) o400->MGC192.168.203.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786414569 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 719.395457] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.124779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.448523] LustreError: 23256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 724.456854] LustreError: 23256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 729.570666] LustreError: 3644:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b44803c8700 x1873190814157312/t0(0) o250->MGC192.168.203.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 729.908073] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 734.291208] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 735.206327] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 735.217730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 735.230673] Lustre: Skipped 1 previous similar message [ 735.279262] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 744.824330] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 750.050508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.058148] Lustre: Skipped 1 previous similar message [ 750.125504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 750.142035] Lustre: Skipped 6 previous similar messages [ 757.955506] Lustre: server umount lustre-MDT0000 complete [ 761.625721] LustreError: 23240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414611 with bad export cookie 54183537859217536 [ 761.635459] LustreError: 23240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 761.643413] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.995938] Lustre: server umount lustre-MDT0001 complete [ 772.319833] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:17:00 (1786414620) [ 792.146730] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 801.299935] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 840.135358] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 851.611888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.887843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 851.920360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 851.984726] Lustre: lustre-MDT0000: new disk, initializing [ 852.056515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 852.061914] Lustre: Skipped 2 previous similar messages [ 852.073093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 855.932611] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 866.845194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.008390] Lustre: 29638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 867.308967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 871.758232] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 881.721722] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 886.800783] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 894.325866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 894.533943] Lustre: 31607:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 896.184976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 896.236239] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 896.346350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 896.365129] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 901.175395] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 910.356463] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 917.369797] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 937.441344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 937.443490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 937.445434] LustreError: Skipped 1 previous similar message [ 937.454919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 937.456136] Lustre: Skipped 2 previous similar messages [ 937.461640] Lustre: Skipped 2 previous similar messages [ 942.568049] Lustre: server umount lustre-OST0000 complete [ 952.801765] LustreError: 29649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.816232] LustreError: 29649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 952.836955] Lustre: server umount lustre-MDT0000 complete [ 956.522179] LustreError: 29630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414806 with bad export cookie 54183537859217942 [ 956.523846] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 956.542517] LustreError: 29630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 956.856839] Lustre: server umount lustre-MDT0001 complete [ 967.004801] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 969.626351] Key type lgssc unregistered [ 969.830927] LNet: 33585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.839783] LNetError: 33585:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.854807] LNet: Removed LNI 192.168.203.117@tcp [ 970.791135] Key type .llcrypt unregistered [ 970.793766] Key type ._llcrypt unregistered [ 996.703370] Key type ._llcrypt registered [ 996.705568] Key type .llcrypt registered [ 996.848440] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 998.247834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.550829] alg: No test for adler32 (adler32-zlib) [ 999.523916] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 999.822337] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 1001.489166] Key type lgssc registered [ 1002.730332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1010.521066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1010.532799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1011.828936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.465972] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1021.413860] LustreError: 34993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1021.436349] LustreError: 34993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1026.007592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.321959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1030.385938] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1037.383695] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1042.495566] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1049.614562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1049.936228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.055077] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1056.189261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1056.693590] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1063.591600] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1077.039694] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:22:04 (1786414924) [ 1079.161366] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1080.881449] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1087.508642] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:22:15 (1786414935) [ 1096.181867] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:22:24 (1786414944) [ 1096.904568] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1107.262408] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1107.265671] Lustre: Skipped 2 previous similar messages [ 1108.964251] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:22:37 (1786414957) [ 1113.190397] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1113.194167] Lustre: Skipped 3 previous similar messages [ 1131.924568] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1131.930802] Lustre: Skipped 6 previous similar messages [ 1134.652483] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:23:02 (1786414982) [ 1137.608672] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1141.393646] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1150.937804] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:23:19 (1786414999) [ 1152.850765] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1152.856751] Lustre: Skipped 3 previous similar messages [ 1166.386484] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:23:34 (1786415014) [ 1180.131442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1180.142634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.177604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1184.225172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1184.228962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1184.267375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.355461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1189.377408] Lustre: Skipped 2 previous similar messages [ 1193.951153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1194.107721] Lustre: server umount lustre-OST0000 complete [ 1199.599277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.602493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1199.608311] Lustre: Skipped 1 previous similar message [ 1199.615333] Lustre: Skipped 1 previous similar message [ 1204.704028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1204.706414] Lustre: Skipped 1 previous similar message [ 1204.887488] Lustre: server umount lustre-MDT0000 complete [ 1209.438501] LustreError: 34974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415059 with bad export cookie 9624102065453892821 [ 1209.440907] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1209.446995] LustreError: 34974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1209.878801] Lustre: server umount lustre-MDT0001 complete [ 1221.197367] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 1223.265772] Key type lgssc unregistered [ 1223.469869] LNet: 42847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.476419] LNetError: 42847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1223.486896] LNet: Removed LNI 192.168.203.117@tcp [ 1224.347125] Key type .llcrypt unregistered [ 1224.352345] Key type ._llcrypt unregistered [ 1245.855714] Key type ._llcrypt registered [ 1245.857807] Key type .llcrypt registered [ 1245.995996] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 1261.051850] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1262.714903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1262.785890] alg: No test for adler32 (adler32-zlib) [ 1263.849766] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 1264.033238] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 1265.687131] Key type lgssc registered [ 1267.052700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1302.869110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1302.894984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.206757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1304.274135] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1304.358040] Lustre: lustre-MDT0000: new disk, initializing [ 1304.524451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1304.558642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1308.951616] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1320.651749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.731209] Lustre: 46001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1320.771904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1320.776644] Lustre: Skipped 1 previous similar message [ 1320.829328] Lustre: lustre-MDT0001: new disk, initializing [ 1320.903321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1320.953053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1320.967213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1325.427475] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1334.369826] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1339.548357] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1346.666213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1346.895203] Lustre: lustre-OST0000: new disk, initializing [ 1346.898852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1346.904678] Lustre: 47970:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1346.994151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.242942] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1353.769485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1353.776683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1353.892476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1362.943457] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1367.436610] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1367.622940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1372.391706] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1372.687533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1374.176432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1374.179944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.187034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1374.708546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1374.739232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.746847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1379.860223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1380.326439] Lustre: server umount lustre-OST0000 complete [ 1384.961758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.962887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1385.088931] Lustre: Skipped 1 previous similar message [ 1385.094683] Lustre: Skipped 1 previous similar message [ 1390.049880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.055741] Lustre: Skipped 2 previous similar messages [ 1390.585813] Lustre: server umount lustre-MDT0000 complete [ 1394.701188] LustreError: 46798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415244 with bad export cookie 10505316380402390537 [ 1394.712799] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1394.723654] LustreError: 46798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1395.387805] Lustre: server umount lustre-MDT0001 complete [ 1407.316562] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:27:35 (1786415255) [ 1415.039228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.560984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.806673] LustreError: 50430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.879934] LustreError: 50430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1421.107133] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1425.892589] LustreError: 50430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.923075] LustreError: 50431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1429.190769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1429.643775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1433.640853] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1440.073454] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1445.925241] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1452.205069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1452.592262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.255753] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1465.261313] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1471.283429] LustreError: 52956:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1472.150317] Lustre: Modifying parameter general.timeout=1 in log params [ 1474.077231] Lustre: Modifying parameter general.timeout=3 in log params [ 1474.080330] Lustre: Skipped 1 previous similar message [ 1476.574613] Lustre: Modifying parameter general.timeout=6 in log params [ 1476.577901] Lustre: Skipped 2 previous similar messages [ 1480.642547] Lustre: Modifying parameter general.timeout=11 in log params [ 1480.646947] Lustre: Skipped 4 previous similar messages [ 1489.026408] Lustre: Modifying parameter general.timeout=22 in log params [ 1489.029244] Lustre: Skipped 10 previous similar messages [ 1505.186822] Lustre: Modifying parameter general.timeout=42 in log params [ 1505.191424] Lustre: Skipped 19 previous similar messages [ 1537.673839] Lustre: Modifying parameter general.timeout=72 in log params [ 1537.680557] Lustre: Skipped 29 previous similar messages [ 1598.970339] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:30:47 (1786415447) [ 1601.708653] Lustre: 59900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1602.635127] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1602.642339] Lustre: Skipped 60 previous similar messages [ 1603.662415] Lustre: 60011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1606.978563] Lustre: 60209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1606.991475] Lustre: 60209:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1642.445433] Lustre: 61778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1642.522878] Lustre: 61778:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1642.565102] LustreError: 61778:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1647.897785] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:31:36 (1786415496) [ 1659.872445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1659.879494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.889775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1664.462321] Lustre: server umount lustre-MDT0000 complete [ 1666.531727] LustreError: 51158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1666.555540] LustreError: 51158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1667.621620] LustreError: 50412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415517 with bad export cookie 10505316380402391412 [ 1667.629391] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1667.636013] LustreError: 50412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1667.847362] Lustre: server umount lustre-MDT0001 complete [ 1681.598813] Lustre: server umount lustre-OST0000 complete [ 1708.425875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.674593] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1708.951831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1713.975337] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1722.687464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.764165] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1722.793159] Lustre: 64310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1727.950572] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1735.349452] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1740.914591] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1749.163868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1749.272833] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1749.500070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.507638] Lustre: Skipped 1 previous similar message [ 1756.072922] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1763.294959] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1765.398750] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1765.408255] Lustre: Skipped 20 previous similar messages [ 1765.507373] Lustre: 66878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1771.496910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.500729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1771.510661] Lustre: Skipped 2 previous similar messages [ 1771.522706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1771.529852] Lustre: Skipped 3 previous similar messages [ 1777.397292] Lustre: server umount lustre-OST0000 complete [ 1781.731957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.740065] Lustre: Skipped 1 previous similar message [ 1786.819301] Lustre: server umount lustre-MDT0000 complete [ 1786.853058] LustreError: 65485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.104656] LustreError: 64304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415640 with bad export cookie 10505316380402394478 [ 1790.118867] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1790.370711] Lustre: server umount lustre-MDT0001 complete [ 1800.493259] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 1802.765086] Key type lgssc unregistered [ 1803.173768] LNet: 68085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.179063] LNetError: 68085:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.188865] LNet: Removed LNI 192.168.203.117@tcp [ 1804.052634] Key type .llcrypt unregistered [ 1804.056901] Key type ._llcrypt unregistered [ 1829.178514] Key type ._llcrypt registered [ 1829.180787] Key type .llcrypt registered [ 1829.261336] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 1830.434450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1830.445872] alg: No test for adler32 (adler32-zlib) [ 1832.036826] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 1832.194826] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 1833.815746] Key type lgssc registered [ 1834.931657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1841.942834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1841.955697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.390055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.987924] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1853.930742] LustreError: 69498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1853.945671] LustreError: 69498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1855.745350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1856.098704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1860.205384] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1867.358911] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1873.260355] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1878.952087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1879.409991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1885.131431] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1892.112739] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1900.485544] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1909.224815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.227432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1909.237818] Lustre: Skipped 2 previous similar messages [ 1909.250050] Lustre: Skipped 1 previous similar message [ 1913.868460] Lustre: server umount lustre-MDT0000 complete [ 1914.340043] LustreError: 69498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1914.349351] LustreError: 69498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1917.329588] LustreError: 69480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415767 with bad export cookie 17818256880353962977 [ 1917.332448] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1917.341903] LustreError: 69480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1917.541625] Lustre: server umount lustre-MDT0001 complete [ 1931.099658] Lustre: server umount lustre-OST0000 complete [ 1955.668462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.848674] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1955.878253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1956.100690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1960.094846] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1967.357477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.416753] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1967.436670] Lustre: 74368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1967.492794] LustreError: 74379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1967.797888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1972.147328] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 1978.242511] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1983.348695] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1989.929254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1990.037748] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1990.415306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.088433] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2002.586461] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2004.305374] Lustre: 76941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2005.778575] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2014.690083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.693024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2014.699663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2014.705672] LustreError: Skipped 1 previous similar message [ 2014.719724] Lustre: Skipped 2 previous similar messages [ 2015.863576] Lustre: server umount lustre-OST0000 complete [ 2019.809299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.812178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.832715] Lustre: Skipped 2 previous similar messages [ 2024.930553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2024.934790] Lustre: Skipped 1 previous similar message [ 2025.458422] Lustre: server umount lustre-MDT0000 complete [ 2028.461643] LustreError: 74359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415878 with bad export cookie 17818256880353963796 [ 2028.462845] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.485149] LustreError: 74359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2028.743188] Lustre: server umount lustre-MDT0001 complete [ 2037.457645] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 2039.627490] Key type lgssc unregistered [ 2039.989951] LNet: 78155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2039.997612] LNetError: 78155:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2040.010955] LNet: Removed LNI 192.168.203.117@tcp [ 2040.617762] Key type .llcrypt unregistered [ 2040.620041] Key type ._llcrypt unregistered [ 2063.584856] Key type ._llcrypt registered [ 2063.586841] Key type .llcrypt registered [ 2063.666211] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 2064.827753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2064.856321] alg: No test for adler32 (adler32-zlib) [ 2065.844672] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 2066.138271] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 2067.791207] Key type lgssc registered [ 2068.734952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2075.040680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2075.060105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2076.346917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.010510] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2086.884996] LustreError: 79568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2086.903944] LustreError: 79568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2087.320572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2087.803555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2091.315920] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2097.648947] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2102.969744] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2108.957346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2109.581216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.380503] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2122.966689] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2129.092290] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2130.280789] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2131.291248] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2133.810494] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2133.815409] Lustre: Skipped 2 previous similar messages [ 2137.945068] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2137.948400] Lustre: Skipped 4 previous similar messages [ 2146.370651] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2146.380394] Lustre: Skipped 8 previous similar messages [ 2162.997248] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2163.002268] Lustre: Skipped 17 previous similar messages [ 2195.199723] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2195.206326] Lustre: Skipped 35 previous similar messages [ 2224.027739] Lustre: 87670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2226.950693] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:41:14 (1786416074) [ 2229.444323] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2231.788369] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:41:19 (1786416079) [ 2235.362473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2235.375185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.399382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2239.462708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2239.464348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.482347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2244.606520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2244.614555] Lustre: Skipped 2 previous similar messages [ 2249.183121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2249.269744] Lustre: server umount lustre-OST0000 complete [ 2254.835038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.836965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2254.842959] Lustre: Skipped 1 previous similar message [ 2254.849389] Lustre: Skipped 1 previous similar message [ 2258.572636] Lustre: server umount lustre-MDT0000 complete [ 2259.942605] LustreError: 80862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.959764] LustreError: 80862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2262.490302] LustreError: 79550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416112 with bad export cookie 2605170143050833558 [ 2262.504485] LustreError: 79550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2262.505407] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.784258] Lustre: server umount lustre-MDT0001 complete [ 2272.633613] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 2276.014821] Key type lgssc unregistered [ 2276.326769] LNet: 88921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.333819] LNetError: 88921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2277.358527] LNet: Removed LNI 192.168.203.117@tcp [ 2278.172957] Key type .llcrypt unregistered [ 2278.175062] Key type ._llcrypt unregistered [ 2283.008833] Key type ._llcrypt registered [ 2283.010523] Key type .llcrypt registered [ 2283.082898] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2288.131804] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules [ 2289.595478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2289.741146] alg: No test for adler32 (adler32-zlib) [ 2290.937510] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 2291.100198] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 2292.719343] Key type lgssc registered [ 2293.791096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2295.650526] LustreError: 90126:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2311.535973] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 2317.750792] LustreError: 90126:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2321.216845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2321.234766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.581807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.041487] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2332.129768] LustreError: 91150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.144848] LustreError: 91150:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2335.281691] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:43:03 (1786416183) [ 2337.258784] LustreError: 91151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.565799] LustreError: 91182:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2339.573453] LustreError: 91182:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2339.580504] LustreError: 91182:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2339.869447] Lustre: server umount lustre-MDT0000 complete [ 2352.861990] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 2355.218247] Key type lgssc unregistered [ 2355.419782] LNet: 92675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.444895] LNetError: 92675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.461879] LNet: Removed LNI 192.168.203.117@tcp [ 2356.034683] Key type .llcrypt unregistered [ 2356.054977] Key type ._llcrypt unregistered [ 2374.589653] Key type ._llcrypt registered [ 2374.593486] Key type .llcrypt registered [ 2374.923348] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 2376.545793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2376.560916] alg: No test for adler32 (adler32-zlib) [ 2378.036256] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 2378.214025] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 2379.879165] Key type lgssc registered [ 2381.091140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2389.305926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2389.345303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.893594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2395.942744] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2400.282460] LustreError: 93896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2400.319857] LustreError: 93896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2404.942241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.532100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2410.812155] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2418.318835] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2424.102699] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2431.077347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2431.845232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.341997] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2443.526720] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2493.399686] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:45:41 (1786416341) [ 2495.126447] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2496.975954] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:45:44 (1786416344) [ 2502.626862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2502.635899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2502.647956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2504.673050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2504.685039] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.053625] Lustre: server umount lustre-MDT0000 complete [ 2509.796904] LustreError: 93895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.808359] LustreError: 93895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2511.117137] LustreError: 93876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416361 with bad export cookie 10330953162838699054 [ 2511.128652] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2511.130108] LustreError: 93876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2511.397382] Lustre: server umount lustre-MDT0001 complete [ 2524.469390] Lustre: server umount lustre-OST0000 complete [ 2536.125530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.435070] LustreError: 98168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.984487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2541.761959] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2542.057118] LustreError: 98169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.152172] LustreError: 98168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.026184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.641029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2555.175884] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2563.080186] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2570.096384] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2583.977037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2584.112725] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2584.118263] LustreError: 98155:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2584.135371] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2584.144266] LustreError: 100253:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2584.149855] LustreError: 100253:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2584.153841] LustreError: 100253:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2584.198543] Lustre: server umount lustre-OST0000 complete [ 2584.200668] LustreError: 100253:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2593.994951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2594.263720] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2594.273572] LustreError: 98155:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2594.291583] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2594.301819] LustreError: 100717:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2594.306756] LustreError: 100717:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2594.311282] LustreError: 100717:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2594.355560] Lustre: server umount lustre-OST0000 complete [ 2594.361747] LustreError: 100717:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2606.669654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2606.787455] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2606.792832] Lustre: Found index 0 for lustre-OST0000, updating log [ 2606.798175] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2606.853607] Lustre: lustre-OST0000: new disk, initializing [ 2606.860235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2606.867314] Lustre: 101254:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2607.113809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2614.069583] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2618.131646] Lustre: Failing over lustre-OST0000 [ 2618.219507] Lustre: server umount lustre-OST0000 complete [ 2625.574455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2625.580445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2625.591792] Lustre: Skipped 2 previous similar messages [ 2625.731219] Lustre: Skipped 3 previous similar messages [ 2630.627141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2630.643543] Lustre: Skipped 1 previous similar message [ 2635.745322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2635.750449] Lustre: Skipped 1 previous similar message [ 2638.303828] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2638.625981] Lustre: server umount lustre-MDT0000 complete [ 2640.877708] LustreError: 98168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.902564] LustreError: 98168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2642.855948] LustreError: 98148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416492 with bad export cookie 10330953162838701805 [ 2642.875957] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2642.886406] LustreError: 98148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2643.337851] Lustre: server umount lustre-MDT0001 complete [ 2652.054168] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:48:20 (1786416500) [ 2667.025532] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 2678.460409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.727489] LustreError: 103600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.829102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.934740] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2690.778178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.283995] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2698.550601] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2705.644936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2713.922384] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2723.924373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2724.013091] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2724.105238] Lustre: lustre-OST0001: new disk, initializing [ 2724.115296] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2724.128113] Lustre: 105919:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2724.559187] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2724.586161] Lustre: Skipped 2 previous similar messages [ 2729.572395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2729.587075] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2729.672778] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2730.218832] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2737.983563] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2744.446932] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2747.874746] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2747.878309] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.884696] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2753.851187] Lustre: server umount lustre-MDT0001 complete [ 2755.554053] LustreError: 103597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.565311] LustreError: 103597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2762.308078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.366221] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2762.373172] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2762.430213] Lustre: 103589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2764.039779] LustreError: 107458:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2764.048458] LustreError: 107458:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2764.062284] LustreError: 103592:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2764.249211] LustreError: 107468:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.117@tcp: cfg command failed: rc = -17 [ 2764.261915] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2764.261915] [ 2767.899049] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 2768.366030] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2777.020889] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:50:24 (1786416624) [ 2792.739729] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 2809.577600] Lustre: 109534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2832.024317] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2835.902684] Lustre: 110669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2921.440578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2921.449466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.462095] Lustre: Skipped 2 previous similar messages [ 2921.465608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.481695] Lustre: Skipped 5 previous similar messages [ 2927.613522] Lustre: server umount lustre-MDT0000 complete [ 2930.144595] LustreError: 106400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2930.161258] LustreError: 106400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2931.304345] LustreError: 105918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416781 with bad export cookie 10330953162838702799 [ 2931.305156] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2931.319532] LustreError: 105918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2931.837840] Lustre: server umount lustre-MDT0001 complete [ 2946.527322] Lustre: server umount lustre-OST0000 complete [ 2959.314102] Lustre: server umount lustre-OST0001 complete [ 2965.777771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2975.123635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2983.575634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2993.584293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3012.327442] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3023.962070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3023.977378] Lustre: lustre-MDT0000: reset Object Index mappings [ 3024.393152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.402835] Lustre: Skipped 1 previous similar message [ 3028.249227] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3036.676675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.691975] Lustre: lustre-MDT0001: reset Object Index mappings [ 3041.989942] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3044.385714] Lustre: 115769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3050.478248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3057.034481] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3061.224815] LustreError: 116122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3061.250836] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3061.263962] LustreError: 116122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3064.793046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3069.514639] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3069.516687] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3069.522072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3071.245765] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3078.841455] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3083.147763] Lustre: 117644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3137.992562] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:56:25 (1786416985) [ 3165.665532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3165.679823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.698549] Lustre: Skipped 3 previous similar messages [ 3165.705256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3165.709050] Lustre: Skipped 4 previous similar messages [ 3169.590570] Lustre: server umount lustre-MDT0000 complete [ 3172.852984] LustreError: 114600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417022 with bad export cookie 10330953162838749650 [ 3172.858652] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.915699] LustreError: 114600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3179.445432] Lustre: server umount lustre-MDT0001 complete [ 3189.591207] Lustre: server umount lustre-OST0000 complete [ 3198.874697] Lustre: server umount lustre-OST0001 complete [ 3205.489269] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 3214.830688] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3251.063317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3251.246799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3251.267090] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3251.321704] Lustre: lustre-MDT0000: new disk, initializing [ 3251.383469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.385876] Lustre: Skipped 3 previous similar messages [ 3251.401598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3256.761048] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3263.719269] Lustre: server umount lustre-MDT0000 complete [ 3273.652167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.974050] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3273.993733] Lustre: 121915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3274.011670] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3278.912962] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3285.498577] Lustre: server umount lustre-MDT0000 complete [ 3297.895984] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:59:05 (1786417145) [ 3299.678615] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3301.755259] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:59:09 (1786417149) [ 3320.219605] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 3331.821759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.219320] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3344.695837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3344.789281] Lustre: 123965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3344.831498] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3344.833807] Lustre: Skipped 1 previous similar message [ 3344.885501] Lustre: lustre-MDT0001: new disk, initializing [ 3345.147696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3345.153134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3349.559992] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3354.319570] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3360.504392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3360.711954] Lustre: lustre-OST0000: new disk, initializing [ 3360.717259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3360.724566] Lustre: 125545:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3362.464407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3362.470635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3362.570280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3366.883991] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3377.148411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3377.347386] Lustre: lustre-OST0001: new disk, initializing [ 3377.350340] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3377.355064] Lustre: 126415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3378.651859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3378.672350] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3378.933153] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3383.872149] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 3393.117454] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3397.453352] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3848.098725] Lustre: 132854:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3848.117353] Lustre: 132854:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3848.141115] Lustre: 132854:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3848.164330] Lustre: 132854:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3848.185787] Lustre: 132854:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3848.204542] Lustre: 132854:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4180.205747] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:13:47 (1786418027) [ 4194.823711] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.839407] Lustre: Skipped 8 previous similar messages [ 4194.846844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4194.861399] Lustre: Skipped 11 previous similar messages [ 4200.848432] Lustre: server umount lustre-MDT0000 complete [ 4204.588037] LustreError: 132868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418054 with bad export cookie 10330953162838818264 [ 4204.592800] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4204.649916] LustreError: 132868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4205.026476] LustreError: 127229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4205.030096] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.034383] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4205.034391] Lustre: Skipped 4 previous similar messages [ 4205.045833] LustreError: 127229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4205.091348] Lustre: Skipped 1 previous similar message [ 4211.048530] Lustre: server umount lustre-MDT0001 complete [ 4221.329859] Lustre: server umount lustre-OST0000 complete [ 4239.327140] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4239.518766] Lustre: server umount lustre-OST0001 complete [ 4245.887948] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 4256.172944] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 4293.241462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4293.512317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4293.565053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4293.717446] Lustre: lustre-MDT0000: new disk, initializing [ 4293.829923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4293.834252] Lustre: Skipped 5 previous similar messages [ 4293.845832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4297.950683] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4309.300865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4309.393480] Lustre: 135921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4309.444290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4309.448572] Lustre: Skipped 1 previous similar message [ 4309.508443] Lustre: lustre-MDT0001: new disk, initializing [ 4309.829956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4309.847082] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4314.478269] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4324.130823] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4329.009468] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4334.917934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4335.143538] Lustre: lustre-OST0000: new disk, initializing [ 4335.146262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4335.155486] Lustre: 137890:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4335.270840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4335.275269] Lustre: Skipped 1 previous similar message [ 4336.563457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4336.576187] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4336.638537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4341.588309] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 4352.595448] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4363.644419] Lustre: lustre-MDD0000: changelog on [ 4366.130882] Lustre: lustre-MDD0001: changelog on [ 6295.974938] Lustre: lustre-MDD0001: changelog off [ 6297.850033] Lustre: 141774:0:(llog_cat.c:984:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6298.899992] Lustre: lustre-MDD0000: changelog off [ 6307.815702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6307.817622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6307.824676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6307.824689] Lustre: Skipped 3 previous similar messages [ 6307.851461] Lustre: Skipped 1 previous similar message [ 6310.667216] Lustre: server umount lustre-OST0000 complete [ 6318.050709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6318.050864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6318.081741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6318.093432] Lustre: Skipped 2 previous similar messages [ 6323.173782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6323.182813] Lustre: Skipped 1 previous similar message [ 6329.311212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6329.709741] Lustre: server umount lustre-MDT0000 complete [ 6332.393224] LustreError: 135928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.399683] LustreError: 135928:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6333.483701] LustreError: 135912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420183 with bad export cookie 10330953162839649248 [ 6333.487486] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.489677] LustreError: 135912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6333.878723] Lustre: server umount lustre-MDT0001 complete [ 6346.949742] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 6349.706562] Key type lgssc unregistered [ 6349.939994] LNet: 147055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6349.959881] LNetError: 147055:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6349.979723] LNet: Removed LNI 192.168.203.117@tcp [ 6351.169711] Key type .llcrypt unregistered [ 6351.174744] Key type ._llcrypt unregistered [ 6372.540341] Key type ._llcrypt registered [ 6372.543586] Key type .llcrypt registered [ 6372.692732] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 6385.514569] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 6386.885941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6386.914713] alg: No test for adler32 (adler32-zlib) [ 6387.884668] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 6388.061208] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 6389.751297] Key type lgssc registered [ 6391.406089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6422.921443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6422.942642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6424.131891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6424.182672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6424.247707] Lustre: lustre-MDT0000: new disk, initializing [ 6424.347924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6424.367925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6428.084122] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 6437.227824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6437.348828] Lustre: 150234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6437.427214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6437.431874] Lustre: Skipped 1 previous similar message [ 6437.496406] Lustre: lustre-MDT0001: new disk, initializing [ 6437.599133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6437.725078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6437.743431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6442.059235] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 6451.620254] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6457.137085] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6464.344596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6464.705777] Lustre: lustre-OST0000: new disk, initializing [ 6464.709500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6464.714714] Lustre: 152207:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6465.003313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6472.325269] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 6472.844278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6472.854382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6473.103533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6482.233729] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6488.759919] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_module obdecho/obdecho [ 6490.915711] LustreError: 153211:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6490.922116] LustreError: 153211:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6493.022809] LustreError: 153307:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6504.421803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6504.436825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6504.462747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6509.025735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6509.026052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.055019] Lustre: Skipped 1 previous similar message [ 6510.455631] Lustre: server umount lustre-MDT0000 complete [ 6514.553482] LustreError: 151030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420364 with bad export cookie 3906368228700930022 [ 6514.562572] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6514.567219] LustreError: 151030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6514.883850] Lustre: server umount lustre-MDT0001 complete [ 6529.164829] Lustre: server umount lustre-OST0000 complete [ 6550.588915] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 6559.264844] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 6595.578058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6595.804399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6595.833130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6596.010627] Lustre: lustre-MDT0000: new disk, initializing [ 6596.136460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6596.160872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6601.304369] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 6611.851534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.928313] Lustre: 157021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6612.006188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6612.012613] Lustre: Skipped 1 previous similar message [ 6612.093868] Lustre: lustre-MDT0001: new disk, initializing [ 6612.202755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6612.353398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6612.445757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6616.862485] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 6625.637985] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6632.051329] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6638.865375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6639.138928] Lustre: lustre-OST0000: new disk, initializing [ 6639.159613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6639.167730] Lustre: 158992:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6639.399873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6646.815953] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 6646.863543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6646.873909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6647.018382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6656.176813] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6661.896240] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6662.336423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6667.969298] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6668.173304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6672.872077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6672.872852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6672.903505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6672.903518] Lustre: Skipped 2 previous similar messages [ 6672.904709] Lustre: Skipped 1 previous similar message [ 6672.936821] LustreError: Skipped 1 previous similar message [ 6677.991997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6678.007226] Lustre: Skipped 2 previous similar messages [ 6683.107698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6683.118968] Lustre: Skipped 1 previous similar message [ 6684.127231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6684.242750] Lustre: server umount lustre-OST0000 complete [ 6689.765120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6689.769953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6693.345362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6693.349671] Lustre: Skipped 2 previous similar messages [ 6694.922588] Lustre: server umount lustre-MDT0000 complete [ 6698.466683] LustreError: 157030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6698.514091] LustreError: 157030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6699.152149] LustreError: 157013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420549 with bad export cookie 3906368228700930981 [ 6699.155877] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6699.158447] LustreError: 157013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6699.553269] Lustre: server umount lustre-MDT0001 complete [ 6703.347813] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:55:51 (1786420551) [ 6706.167514] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6707.964714] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:55:55 (1786420555) [ 6716.846604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.378751] LustreError: 161406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.524562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6722.544281] LustreError: 161407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6723.954911] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 6727.655478] LustreError: 161406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6731.821483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6736.070870] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 6742.753701] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6748.157713] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6755.264340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6755.485571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6755.493036] Lustre: Skipped 1 previous similar message [ 6760.913940] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 6767.585581] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6773.134381] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.17@tcp (stopping) [ 6773.222045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6773.228368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6773.236945] Lustre: Skipped 3 previous similar messages [ 6775.468791] Lustre: server umount lustre-MDT0000 complete [ 6778.214464] LustreError: 161403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6778.844145] LustreError: 161387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420628 with bad export cookie 3906368228700931863 [ 6778.845925] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6778.857457] LustreError: 161387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6783.458477] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.447120] LustreError: 161403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6788.463378] LustreError: 161403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6793.186117] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6793.432372] Lustre: server umount lustre-MDT0001 complete [ 6799.200429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6811.758650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6811.895338] LustreError: 165250:0:(llog.c:1655:llog_backup()) MGC192.168.203.117@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6811.899186] Lustre: 165250:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.117@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6814.119500] LustreError: 165255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6814.140095] LustreError: 165255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6849.895766] LustreError: 165257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6849.912499] LustreError: 165257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 6872.543475] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6872.554670] Lustre: 165250:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.117@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6913.609864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6914.400678] LustreError: 165256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6914.435905] LustreError: 165256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 43 previous similar messages [ 6918.190573] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 6918.640490] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6928.020684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6941.153674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6950.367724] LustreError: 148458:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f28e17c1 x1873197387617792/t0(0) o250->MGC192.168.203.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6955.958816] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 6956.019158] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6963.241526] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6965.124616] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6970.656861] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6973.519204] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6991.870051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6991.875770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6991.892709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6991.904752] Lustre: Skipped 10 previous similar messages [ 7005.168381] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7005.367653] Lustre: server umount lustre-MDT0000 complete [ 7008.689335] LustreError: 161387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420858 with bad export cookie 3906368228700938352 [ 7008.702335] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7008.704237] LustreError: 161387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7009.131914] Lustre: server umount lustre-MDT0001 complete [ 7023.282961] Lustre: server umount lustre-OST0000 complete [ 7043.479612] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 7053.912450] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 7089.692101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7089.967043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7090.078815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7090.131155] Lustre: lustre-MDT0000: new disk, initializing [ 7090.236104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7090.251417] Lustre: Skipped 1 previous similar message [ 7090.285483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7095.333495] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7105.340027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7105.477469] Lustre: 170374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7105.566811] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7105.581810] Lustre: Skipped 6 previous similar messages [ 7106.153845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7110.356445] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7120.362244] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7127.470581] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7135.052990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7135.298288] Lustre: lustre-OST0000: new disk, initializing [ 7135.301668] Lustre: Skipped 1 previous similar message [ 7135.305360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7135.309182] Lustre: Skipped 2 previous similar messages [ 7135.314775] Lustre: 172343:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7141.362601] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7143.455368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7143.460055] Lustre: Skipped 1 previous similar message [ 7143.462801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7143.587354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7150.929921] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7156.729779] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7156.906931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7161.516241] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7161.812229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7163.872897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7163.880831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7163.890428] Lustre: Skipped 2 previous similar messages [ 7169.885706] Lustre: server umount lustre-OST0000 complete [ 7179.248499] Lustre: server umount lustre-MDT0000 complete [ 7179.756160] LustreError: 170382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7179.783428] LustreError: 170382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 7182.828420] LustreError: 170365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421032 with bad export cookie 3906368228700955152 [ 7182.838579] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7182.845884] LustreError: 170365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7183.198954] Lustre: server umount lustre-MDT0001 complete [ 7191.997947] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:04:00 (1786421040) [ 7199.675728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7204.599492] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7213.598466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7218.347137] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7225.809889] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7232.648179] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7239.006279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7246.838403] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7254.681654] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7268.888818] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:05:16 (1786421116) [ 7274.473825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7274.479249] LustreError: Skipped 1 previous similar message [ 7274.482702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.488822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7274.492817] Lustre: Skipped 4 previous similar messages [ 7274.503389] Lustre: Skipped 7 previous similar messages [ 7280.066533] Lustre: server umount lustre-OST0000 complete [ 7289.345936] Lustre: server umount lustre-MDT0000 complete [ 7293.410650] LustreError: 174736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421143 with bad export cookie 3906368228700956034 [ 7293.418995] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7293.841654] Lustre: server umount lustre-MDT0001 complete [ 7306.060371] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 7309.025505] Key type lgssc unregistered [ 7309.338410] LNet: 178481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7309.342872] LNetError: 178481:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7310.386101] LNet: Removed LNI 192.168.203.117@tcp [ 7311.082105] Key type .llcrypt unregistered [ 7311.084305] Key type ._llcrypt unregistered [ 7329.766230] Key type ._llcrypt registered [ 7329.768569] Key type .llcrypt registered [ 7329.854490] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 7331.745753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7331.801332] alg: No test for adler32 (adler32-zlib) [ 7332.858384] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 7333.238301] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 7335.079294] Key type lgssc registered [ 7337.319720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7344.456989] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7344.468994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7360.031625] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7365.151849] LustreError: 179728:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.117@tcp: failed processing log, type 4: rc = -110 [ 7370.289767] LustreError: 179728:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7370.302543] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7370.305691] LustreError: MGC192.168.203.117@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7370.314846] LustreError: 179728:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7370.320263] LustreError: 179728:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7370.325188] LustreError: 179728:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7370.358948] Lustre: server umount lustre-OST0000 complete [ 7370.364324] LustreError: 179728:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7379.059173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.397186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7384.635923] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7390.689724] LustreError: 180229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7390.722519] LustreError: 180229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7395.096696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7395.508034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7399.969397] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7407.209915] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7413.872708] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7421.768914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7422.127973] LustreError: 182044:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7422.135371] Lustre: 182044:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.117@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7422.283094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7428.143793] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7436.677835] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7441.380800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7441.394729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7441.449771] Lustre: Skipped 2 previous similar messages [ 7441.475548] Lustre: Skipped 2 previous similar messages [ 7445.620950] Lustre: server umount lustre-MDT0000 complete [ 7446.519724] LustreError: 180224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7446.545907] LustreError: 180224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7449.016825] LustreError: 180208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421298 with bad export cookie 16058459080335219048 [ 7449.025752] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7449.031973] LustreError: 180208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7449.695261] LustreError: 180957:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003d60deb5 x1873198378413696/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7449.870670] Lustre: server umount lustre-MDT0001 complete [ 7463.930233] Lustre: server umount lustre-OST0000 complete [ 7474.197242] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 7485.308532] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 7523.298900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7523.523991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7523.572770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7523.675815] Lustre: lustre-MDT0000: new disk, initializing [ 7523.761994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7523.776268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7528.224741] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7539.474222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7539.574656] Lustre: 185566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7539.617939] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7539.621295] Lustre: Skipped 1 previous similar message [ 7539.678655] Lustre: lustre-MDT0001: new disk, initializing [ 7539.767738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7539.818765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7539.831271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7544.646461] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7552.961408] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7558.704810] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7565.708247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7565.898557] Lustre: lustre-OST0000: new disk, initializing [ 7565.902806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7565.908683] Lustre: 187540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7565.960570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7571.916292] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7576.099950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7576.108277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7576.183336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7581.571299] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7586.094837] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7586.694386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7591.082302] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7591.344722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7596.513951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7596.544119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7596.544408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7596.566703] Lustre: Skipped 1 previous similar message [ 7599.769593] Lustre: server umount lustre-OST0000 complete [ 7606.241317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7606.246886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7606.255013] Lustre: Skipped 1 previous similar message [ 7606.259548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7609.985670] Lustre: server umount lustre-MDT0000 complete [ 7611.875372] LustreError: 185580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7611.898404] LustreError: 185580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7613.395066] LustreError: 185561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421463 with bad export cookie 16058459080335219769 [ 7613.405379] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7613.407075] LustreError: 185561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7613.774591] Lustre: server umount lustre-MDT0001 complete [ 7622.947383] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:11:10 (1786421470) [ 7639.204639] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 7642.921265] Key type lgssc unregistered [ 7643.137550] LNet: 190485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7643.148689] LNetError: 190485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7643.160771] LNet: Removed LNI 192.168.203.117@tcp [ 7643.810731] Key type .llcrypt unregistered [ 7643.813795] Key type ._llcrypt unregistered [ 7662.332772] Key type ._llcrypt registered [ 7662.335185] Key type .llcrypt registered [ 7662.440920] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 7663.252837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7663.266917] alg: No test for adler32 (adler32-zlib) [ 7664.367169] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 7664.584883] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 7666.264082] Key type lgssc registered [ 7667.507694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7674.848789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7674.860163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7676.179773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7680.185606] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7686.624604] LustreError: 191704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7686.635494] LustreError: 191704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7688.634866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.881280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7693.915521] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7701.114217] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7706.579887] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7713.194710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7713.349320] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7713.352213] LustreError: 193569:0:(llog.c:1696:llog_backup()) MGC192.168.203.117@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7713.361273] Lustre: 193569:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.117@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7713.506763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7719.129750] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7725.326930] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7730.170780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7730.174966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7730.181601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7732.212195] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.214582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7732.221182] Lustre: Skipped 1 previous similar message [ 7732.230112] Lustre: Skipped 1 previous similar message [ 7733.687775] Lustre: server umount lustre-MDT0000 complete [ 7736.742424] LustreError: 191686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421586 with bad export cookie 16032029828543368925 [ 7736.750427] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7736.756604] LustreError: 191686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7737.023429] Lustre: server umount lustre-MDT0001 complete [ 7750.415778] Lustre: server umount lustre-OST0000 complete [ 7760.399708] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 7771.706693] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 7806.191323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.388467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7806.413250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7806.474569] Lustre: lustre-MDT0000: new disk, initializing [ 7806.533490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7806.550895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7810.412391] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7820.541531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7820.635773] Lustre: 197094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7820.702588] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7820.704769] Lustre: Skipped 1 previous similar message [ 7820.707172] LustreError: 197875:0:(llog.c:1696:llog_backup()) MGC192.168.203.117@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7820.712280] LustreError: 197875:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7820.715870] Lustre: 197875:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.117@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7820.729132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7820.732954] Lustre: Skipped 1 previous similar message [ 7820.785397] Lustre: lustre-MDT0001: new disk, initializing [ 7820.874644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7820.908281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7820.923177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7825.485079] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7834.066457] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7839.364088] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7845.646495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7845.854905] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7845.856879] Lustre: Skipped 1 previous similar message [ 7845.859888] LustreError: 199038:0:(llog.c:1696:llog_backup()) MGC192.168.203.117@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7845.866568] LustreError: 199038:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7845.871514] Lustre: 199038:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.117@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7845.878907] Lustre: 199038:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7845.900389] Lustre: lustre-OST0000: new disk, initializing [ 7845.904204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7845.909367] Lustre: 199061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7845.987194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7851.360800] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7854.151863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7854.166711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7854.274894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7861.236892] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7866.425633] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7866.639289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7870.585302] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7870.737454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7875.079293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7875.085921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7875.097844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7875.107718] LustreError: Skipped 1 previous similar message [ 7875.176664] Lustre: Skipped 1 previous similar message [ 7878.868986] Lustre: server umount lustre-OST0000 complete [ 7882.211208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7882.230412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7882.250720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7882.265614] Lustre: Skipped 1 previous similar message [ 7888.741793] Lustre: server umount lustre-MDT0000 complete [ 7890.404445] LustreError: 197942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7890.424316] LustreError: 197942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7893.599261] LustreError: 197084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421743 with bad export cookie 16032029828543369646 [ 7893.602132] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7893.630064] LustreError: 197084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7894.326893] Lustre: server umount lustre-MDT0001 complete [ 7904.591370] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:15:52 (1786421752) [ 7920.656507] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 7932.810496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7933.081070] LustreError: 202189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.133325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7937.762033] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7938.534686] LustreError: 202190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7943.652626] LustreError: 202189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7946.959557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7953.158461] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7956.637621] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7963.586104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7970.180954] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7980.460229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7980.650534] Lustre: lustre-OST0001: new disk, initializing [ 7980.654549] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7980.660177] Lustre: 204498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7980.778046] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7980.785587] Lustre: Skipped 2 previous similar messages [ 7988.281309] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 7989.917520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7989.942551] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7990.261520] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7999.978607] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8007.150286] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8010.503785] Lustre: DEBUG MARKER: ADD OST3 [ 8015.794913] loop: module loaded [ 8015.948717] loop0: detected capacity change from 0 to 204800000 [ 8016.442447] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8016.454984] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8016.502485] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8024.741650] loop0: detected capacity change from 0 to 204800000 [ 8024.770075] Lustre: DEBUG MARKER: STOP OST3 [ 8036.496419] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8036.755740] Lustre: lustre-OST0003: new disk, initializing [ 8036.759222] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8036.764573] Lustre: 206555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8038.546964] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8038.761147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8038.778481] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8038.872509] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8040.608622] LustreError: 206638:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8041.762030] Lustre: Failing over lustre-OST0003 [ 8042.007299] Lustre: server umount lustre-OST0003 complete [ 8043.956481] LustreError: 203672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8044.025762] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8044.043126] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8044.106320] Lustre: Skipped 1 previous similar message [ 8049.152270] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8064.350145] LustreError: 204933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8064.373137] LustreError: 204933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 8064.580239] Lustre: DEBUG MARKER: START OST3 again [ 8071.952457] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8072.364159] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8072.391357] Lustre: Skipped 1 previous similar message [ 8072.418938] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8073.704726] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8074.090488] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8074.099228] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8079.600974] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8102.077874] Lustre: Failing over lustre-OST0003 [ 8102.128651] Lustre: server umount lustre-OST0003 complete [ 8103.413326] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8103.419740] LustreError: 203672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8103.440582] Lustre: Skipped 2 previous similar messages [ 8103.464261] LustreError: 203672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8122.355340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8122.366195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8122.401218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8122.406626] Lustre: Skipped 2 previous similar messages [ 8125.922713] Lustre: server umount lustre-MDT0000 complete [ 8129.895530] LustreError: 202170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421979 with bad export cookie 16032029828543370528 [ 8129.902069] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8129.914892] LustreError: 202170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8134.121899] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8134.130158] Lustre: Skipped 4 previous similar messages [ 8144.351149] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8144.585591] Lustre: server umount lustre-MDT0001 complete [ 8149.154380] Lustre: server umount lustre-OST0000 complete [ 8153.297966] Lustre: server umount lustre-OST0001 complete [ 8159.686467] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 8171.110074] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 8210.352985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8210.917926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8211.191574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8211.239933] Lustre: lustre-MDT0000: new disk, initializing [ 8211.326818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8211.355378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8215.490713] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8226.614925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8226.753349] Lustre: 212118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8227.130697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8231.374472] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8242.159154] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8248.553408] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8257.085389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8257.494926] Lustre: lustre-OST0000: new disk, initializing [ 8257.498472] Lustre: Skipped 1 previous similar message [ 8257.501945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8257.506333] Lustre: Skipped 2 previous similar messages [ 8257.511679] Lustre: 214085:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8259.310390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8259.318501] Lustre: Skipped 1 previous similar message [ 8259.322225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8259.482685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8264.651722] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8273.713867] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8278.369664] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8278.671706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8284.274705] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8284.700309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8290.292248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8290.293182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8290.322752] Lustre: Skipped 5 previous similar messages [ 8290.331673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8290.331680] Lustre: Skipped 4 previous similar messages [ 8293.385223] Lustre: server umount lustre-OST0000 complete [ 8312.287085] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8312.445436] Lustre: server umount lustre-MDT0000 complete [ 8316.250099] LustreError: 212110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422166 with bad export cookie 16032029828543372810 [ 8316.259948] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8316.265589] LustreError: 212110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8316.523251] Lustre: server umount lustre-MDT0001 complete [ 8320.687876] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:22:48 (1786422168) [ 8337.474441] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 8348.663128] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 8393.291715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8393.646543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8393.670031] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8393.804865] Lustre: lustre-MDT0000: new disk, initializing [ 8393.930360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8398.860665] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8409.803494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8409.945930] Lustre: 218698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8410.239689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8414.944852] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8425.767938] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8431.425255] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8438.294215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8438.645607] Lustre: 220669:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8440.089786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8445.839842] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8455.497746] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8460.861570] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8461.035809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8467.013251] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8467.280181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8471.016477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8471.019094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8471.026095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8471.026104] Lustre: Skipped 10 previous similar messages [ 8471.044461] Lustre: Skipped 4 previous similar messages [ 8471.053883] LustreError: Skipped 3 previous similar messages [ 8475.442355] Lustre: server umount lustre-OST0000 complete [ 8485.421500] Lustre: server umount lustre-MDT0000 complete [ 8486.371795] LustreError: 218706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8486.396861] LustreError: 218706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 8488.339024] LustreError: 218691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422338 with bad export cookie 16032029828543373692 [ 8488.352490] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8488.579277] Lustre: server umount lustre-MDT0001 complete [ 8497.861566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8498.309357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8498.318859] Lustre: Skipped 5 previous similar messages [ 8504.207459] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8513.615920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8517.858808] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8525.455987] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8531.924612] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8539.353925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8545.995681] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8555.009250] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8568.300676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8568.314076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8568.323599] Lustre: Skipped 3 previous similar messages [ 8572.162661] Lustre: server umount lustre-OST0000 complete [ 8582.383176] Lustre: server umount lustre-MDT0000 complete [ 8586.003406] LustreError: 222868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422435 with bad export cookie 16032029828543374581 [ 8586.021576] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8586.035422] LustreError: 222868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8586.508595] Lustre: server umount lustre-MDT0001 complete [ 8695.392517] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 8697.775372] Key type lgssc unregistered [ 8698.054226] LNet: 226550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8698.063594] LNetError: 226550:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8698.080234] LNet: Removed LNI 192.168.203.117@tcp [ 8699.056249] Key type .llcrypt unregistered [ 8699.059739] Key type ._llcrypt unregistered [ 8723.758269] Key type ._llcrypt registered [ 8723.761205] Key type .llcrypt registered [ 8723.857476] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 8740.447273] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 8741.779347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8741.803373] alg: No test for adler32 (adler32-zlib) [ 8742.872527] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 8743.132653] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 8744.799602] Key type lgssc registered [ 8745.745514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8779.913783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8779.935430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8781.173726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8781.301538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8781.456460] Lustre: lustre-MDT0000: new disk, initializing [ 8781.524373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8781.538954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8787.038448] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8797.248084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8797.436449] Lustre: 229904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8797.482325] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8797.486272] Lustre: Skipped 1 previous similar message [ 8797.851855] Lustre: lustre-MDT0001: new disk, initializing [ 8798.138813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8798.270469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8798.306970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8803.651845] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8812.016460] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8816.967788] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8823.934823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8824.169332] Lustre: lustre-OST0000: new disk, initializing [ 8824.173893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8824.180285] Lustre: 231873:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8824.284148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8830.057745] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8832.546278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8832.562713] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8832.723813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8840.288149] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8845.424834] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8845.901172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8850.616452] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8850.822968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8853.475574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8853.476736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8853.498167] Lustre: Skipped 1 previous similar message [ 8853.499368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8853.502475] Lustre: Skipped 1 previous similar message [ 8858.595595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8858.600943] Lustre: Skipped 1 previous similar message [ 8858.859374] Lustre: server umount lustre-OST0000 complete [ 8863.726702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8863.739020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8863.742844] Lustre: Skipped 1 previous similar message [ 8863.754528] Lustre: Skipped 1 previous similar message [ 8868.832698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8868.837791] Lustre: Skipped 1 previous similar message [ 8869.306254] Lustre: server umount lustre-MDT0000 complete [ 8874.463317] LustreError: 229896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422724 with bad export cookie 1010772840813952703 [ 8874.470895] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8874.474138] LustreError: 229896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8874.975297] Lustre: server umount lustre-MDT0001 complete [ 8896.644631] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 8908.308451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8908.547160] LustreError: 234809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8908.559958] LustreError: 234809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8908.591519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8913.220720] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8913.897125] LustreError: 234810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8919.008741] LustreError: 234809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8922.048387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8922.413125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8927.342612] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8930.360975] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8935.749615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8936.110428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8942.489810] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8951.138081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8951.452924] Lustre: lustre-OST0001: new disk, initializing [ 8951.457996] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8951.479458] Lustre: 237116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8958.531743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8958.535262] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8958.672715] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8959.261348] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 8969.964543] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8975.939111] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8991.851837] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:33:59 (1786422839) [ 8994.786562] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8994.789639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8994.795640] Lustre: Skipped 1 previous similar message [ 8994.804776] Lustre: Skipped 2 previous similar messages [ 8999.393692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8999.399096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8999.421994] Lustre: Skipped 1 previous similar message [ 9001.046574] Lustre: server umount lustre-MDT0000 complete [ 9005.042944] LustreError: 234804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9005.062229] LustreError: 234804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9005.523287] LustreError: 234789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422855 with bad export cookie 1010772840813953585 [ 9005.530158] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9006.266369] Lustre: server umount lustre-MDT0001 complete [ 9021.946509] Lustre: server umount lustre-OST0000 complete [ 9036.813210] Lustre: server umount lustre-OST0001 complete [ 9044.427287] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 9056.469854] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 9091.759454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9092.109677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9092.131135] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9092.175427] Lustre: lustre-MDT0000: new disk, initializing [ 9092.242771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9092.247860] Lustre: Skipped 1 previous similar message [ 9092.260310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9097.334195] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9108.032720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9108.174944] Lustre: 241064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9108.246847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9108.250329] Lustre: Skipped 1 previous similar message [ 9108.303533] Lustre: lustre-MDT0001: new disk, initializing [ 9108.449074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9108.462778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9113.553477] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9122.732801] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9128.342888] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9134.562036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9134.826231] Lustre: lustre-OST0000: new disk, initializing [ 9134.829330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9134.834633] Lustre: 243034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9134.923297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9134.929248] Lustre: Skipped 1 previous similar message [ 9136.580623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9136.599182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9136.741116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9143.554459] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9155.391664] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9159.721233] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9159.892202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9164.396784] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9164.855860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9167.329780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9167.332320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9167.344672] LustreError: Skipped 1 previous similar message [ 9167.346168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9167.358307] Lustre: Skipped 1 previous similar message [ 9167.386732] Lustre: Skipped 5 previous similar messages [ 9173.055928] Lustre: server umount lustre-OST0000 complete [ 9180.641389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9180.643117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9180.669950] Lustre: Skipped 1 previous similar message [ 9185.770504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9185.780450] Lustre: Skipped 5 previous similar messages [ 9191.391124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9191.567707] Lustre: server umount lustre-MDT0000 complete [ 9195.236113] LustreError: 241057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423045 with bad export cookie 1010772840813955713 [ 9195.238553] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9195.252343] LustreError: 241057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9195.564380] Lustre: server umount lustre-MDT0001 complete [ 9204.785645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9205.077122] LustreError: 245253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9205.132904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9209.253328] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9215.462136] LustreError: 245254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9215.480667] LustreError: 245254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9217.813536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9221.746883] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9228.281875] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9233.648528] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9240.954631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9247.272208] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9253.345644] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9399.267851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9399.278265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9399.292224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9399.295645] Lustre: Skipped 1 previous similar message [ 9399.303546] Lustre: Skipped 2 previous similar messages [ 9403.305546] Lustre: server umount lustre-OST0000 complete [ 9407.464448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9412.621729] Lustre: server umount lustre-MDT0000 complete [ 9414.627663] LustreError: 247778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9414.654586] LustreError: 247778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9415.667595] LustreError: 245237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423265 with bad export cookie 1010772840813956602 [ 9415.675872] LustreError: 245237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9415.677602] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9416.024790] Lustre: server umount lustre-MDT0001 complete [ 9521.501355] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [ 9524.279728] Key type lgssc unregistered [ 9524.520751] LNet: 248873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9524.532413] LNetError: 248873:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9524.546864] LNet: Removed LNI 192.168.203.117@tcp [ 9525.933263] Key type .llcrypt unregistered [ 9525.941913] Key type ._llcrypt unregistered [ 9557.346595] Key type ._llcrypt registered [ 9557.348991] Key type .llcrypt registered [ 9557.432344] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 9572.925132] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 9574.895435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9575.163142] alg: No test for adler32 (adler32-zlib) [ 9576.776435] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [ 9576.962554] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [ 9578.599162] Key type lgssc registered [ 9579.949864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9626.738762] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 9637.781502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9637.814670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9639.394054] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9639.427579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9639.480699] Lustre: lustre-MDT0000: new disk, initializing [ 9639.532880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9639.546517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9644.472033] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9655.800505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9655.879238] Lustre: 252948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9655.932286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9655.936506] Lustre: Skipped 1 previous similar message [ 9656.018212] Lustre: lustre-MDT0001: new disk, initializing [ 9656.152987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9656.199852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9656.333335] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9660.449532] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9665.214510] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9672.557545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9672.814604] Lustre: lustre-OST0000: new disk, initializing [ 9672.832859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9672.851290] Lustre: 254581:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9672.965538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9673.047553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9673.070219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9673.194110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9678.290467] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9687.940789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9688.182224] Lustre: lustre-OST0001: new disk, initializing [ 9688.188275] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9688.193367] Lustre: 255452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9688.311416] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9693.973646] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9695.533473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9695.538887] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9695.575735] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9706.603762] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9713.480409] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9726.452335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9726.458571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9726.467586] Lustre: Skipped 3 previous similar messages [ 9726.475706] Lustre: Skipped 1 previous similar message [ 9731.961310] Lustre: server umount lustre-MDT0000 complete [ 9736.246921] LustreError: 252940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423586 with bad export cookie 1783876185256938355 [ 9736.258465] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9736.263704] LustreError: 252940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9736.460478] Lustre: server umount lustre-MDT0001 complete [ 9750.618929] Lustre: server umount lustre-OST0000 complete [ 9753.055921] Lustre: 250453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786423586/real 1786423586] req@ffff8b446d2b1880 x1873200730418944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786423602 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9753.085899] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9754.267576] Lustre: server umount lustre-OST0001 complete [ 9767.598758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9777.059539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9784.936470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9795.026278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9813.342663] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 9822.371732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9822.386263] Lustre: lustre-MDT0000: reset Object Index mappings [ 9822.668486] LustreError: 260475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9822.696418] LustreError: 260475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9822.753445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9826.468850] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9827.812188] LustreError: 260476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9832.945575] LustreError: 260475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9834.279879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9834.295311] Lustre: lustre-MDT0001: reset Object Index mappings [ 9834.796948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9839.311382] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9841.940751] Lustre: 261615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9848.602364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9855.518328] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9859.242720] LustreError: 261969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9863.509590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9863.789710] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9863.797775] Lustre: Skipped 1 previous similar message [ 9867.928091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9867.959323] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9870.219691] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [ 9878.622843] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9882.531463] Lustre: 263489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9888.740246] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9888.750930] Lustre: Skipped 1 previous similar message [ 9888.753040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9888.768292] Lustre: Skipped 5 previous similar messages [ 9892.990776] Lustre: server umount lustre-MDT0000 complete [ 9893.859065] LustreError: 260471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9893.887205] LustreError: 260471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9897.079851] LustreError: 262797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423747 with bad export cookie 1783876185256945138 [ 9897.147485] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9897.166623] LustreError: 262797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9897.600763] Lustre: server umount lustre-MDT0001 complete [ 9911.491416] Lustre: server umount lustre-OST0000 complete [ 9914.335248] Lustre: 250452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786423748/real 1786423748] req@ffff8b434a5e5500 x1873200730466560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786423764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9914.360586] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9914.373028] Lustre: Skipped 1 previous similar message [ 9915.676775] Lustre: server umount lustre-OST0001 complete [ 9935.416568] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:49:43 (1786423783) [ 9951.046365] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [ 9961.226762] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [ 9994.417841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9994.729982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9994.753604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9994.806454] Lustre: lustre-MDT0000: new disk, initializing [ 9994.858891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9994.878025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9999.251918] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10009.350906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10009.451796] Lustre: 267743:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10009.506950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10009.511478] Lustre: Skipped 1 previous similar message [10009.566569] Lustre: lustre-MDT0001: new disk, initializing [10009.719471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10009.735492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10014.010345] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10021.879736] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10027.578785] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10033.584039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10033.793782] Lustre: lustre-OST0000: new disk, initializing [10033.798925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10033.805163] Lustre: 269709:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10035.660216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10035.674760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10035.761486] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10039.388277] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10047.353772] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10052.191993] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10052.421428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10056.962069] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10057.275146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10061.282751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10061.289549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10061.294292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10061.318323] Lustre: Skipped 1 previous similar message [10065.277277] Lustre: server umount lustre-OST0000 complete [10071.008219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10071.015564] LustreError: Skipped 1 previous similar message [10071.021473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10071.045081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10071.050256] Lustre: Skipped 1 previous similar message [10076.640813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10076.649457] Lustre: Skipped 2 previous similar messages [10082.271146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10082.432306] Lustre: server umount lustre-MDT0000 complete [10085.638770] LustreError: 267735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423935 with bad export cookie 1783876185256947238 [10085.668065] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10085.669436] LustreError: 267735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10086.100582] Lustre: server umount lustre-MDT0001 complete [10102.755714] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [10112.474160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10112.773621] LustreError: 272644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10112.798146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10112.801755] Lustre: Skipped 2 previous similar messages [10117.028953] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10125.126470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10130.214866] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10133.139877] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10138.550083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10144.491481] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10151.239175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10151.559675] Lustre: lustre-OST0001: new disk, initializing [10151.562624] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10151.569135] Lustre: 274953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10152.936155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10152.955567] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10153.025172] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10157.597112] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10166.425546] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10169.277922] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10175.350918] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10175.626914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10181.065849] Lustre: server umount lustre-MDT0000 complete [10181.600192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10181.604512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10181.611429] Lustre: Skipped 1 previous similar message [10181.617582] LustreError: 272645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10181.635435] LustreError: 272645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10183.890631] LustreError: 272625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424033 with bad export cookie 1783876185256948120 [10183.893534] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10183.955416] LustreError: 272625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10186.220434] Lustre: server umount lustre-MDT0001 complete [10199.739808] Lustre: server umount lustre-OST0000 complete [10213.737933] Lustre: server umount lustre-OST0001 complete [10227.274350] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [10236.339600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.534901] LustreError: 278009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10236.558032] LustreError: 278009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10240.413347] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10247.275770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10247.572187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10247.579359] Lustre: Skipped 4 previous similar messages [10251.788363] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10254.396258] Lustre: 279152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10260.607671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10261.930636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10266.778770] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10276.241384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10281.989147] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10282.614917] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10290.744992] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10294.521442] Lustre: 281028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10300.161844] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10300.262451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10310.646520] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:55:58 (1786424158) [10328.040563] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10328.044065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10328.120331] Lustre: Skipped 6 previous similar messages [10328.137418] Lustre: Skipped 5 previous similar messages [10332.044416] Lustre: server umount lustre-MDT0000 complete [10333.157833] LustreError: 278744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10333.218226] LustreError: 278744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [10335.154425] LustreError: 277992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424185 with bad export cookie 1783876185256954490 [10335.165227] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10335.180045] LustreError: 277992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10341.471451] Lustre: server umount lustre-MDT0001 complete [10350.620827] Lustre: server umount lustre-OST0000 complete [10354.002575] Lustre: server umount lustre-OST0001 complete [10360.037693] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [10370.656806] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [10415.222904] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [10425.911832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10426.187475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10426.241179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10426.419274] Lustre: lustre-MDT0000: new disk, initializing [10426.635912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10430.929531] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10439.384411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10439.480707] Lustre: 285266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10439.684900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10444.096234] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10453.766459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10453.944329] Lustre: 286897:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10455.735182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10459.649464] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10469.338347] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10469.512764] Lustre: 287769:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10471.360857] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10476.084645] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10485.951249] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10489.239267] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10489.242963] Lustre: Skipped 1 previous similar message [10501.758716] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:59:10 (1786424350) [10506.208569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10506.217846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10506.232331] Lustre: Skipped 2 previous similar messages [10506.235500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10506.239127] Lustre: Skipped 4 previous similar messages [10506.368180] Lustre: server umount lustre-MDT0000 complete [10507.240919] LustreError: 285278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10507.302389] LustreError: 285278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10510.553354] LustreError: 285259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424360 with bad export cookie 1783876185256962372 [10510.561742] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10510.567263] LustreError: 285259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10510.774470] Lustre: server umount lustre-MDT0001 complete [10524.223861] Lustre: server umount lustre-OST0000 complete [10537.812188] Lustre: server umount lustre-OST0001 complete [10544.517481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10545.092865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10545.101441] Lustre: Skipped 6 previous similar messages [10549.354610] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10556.177864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10560.561539] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10567.465401] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10573.020862] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10580.382868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10586.258911] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10593.619556] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10610.355976] loop1: detected capacity change from 0 to 204800000 [10610.472479] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10610.483435] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10610.504345] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10615.117201] loop1: detected capacity change from 0 to 204800000 [10620.544622] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10620.605417] LustreError: 290005:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs13753: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10620.616279] LustreError: 290005:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs13753 [10620.620551] LustreError: 290005:0:(mgs_handler.c:517:mgs_target_reg()) fs13753-MDT0001: Error checking target index: rc = -13 [10620.626988] LustreError: fs13753-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10620.632968] LustreError: 293751:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10620.639327] LustreError: 293751:0:(tgt_mount.c:2129:server_put_super()) no obd fs13753-MDT0001 [10620.643309] LustreError: 293751:0:(tgt_mount.c:116:server_deregister_mount()) fs13753-MDT0001 not registered [10620.671834] Lustre: server umount fs13753-MDT0001 complete [10620.674267] LustreError: 293751:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10628.914532] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10628.977410] Lustre: Setting parameter fs13753-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13753-MDT0001 [10629.006585] Lustre: srv-fs13753-MDT0001: No data found on store. Initialize space. [10629.013656] Lustre: Skipped 4 previous similar messages [10629.062317] Lustre: fs13753-MDT0001: new disk, initializing [10629.065022] Lustre: Skipped 3 previous similar messages [10629.329957] Lustre: srv-fs13753-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10629.345838] Lustre: 294217:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13753-MDT0001-mdtlov: can't create new llog: rc = -115 [10629.368864] LustreError: 294217:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13753-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10633.782954] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10640.922435] Lustre: server umount fs13753-MDT0001 complete [10653.780850] loop1: detected capacity change from 0 to 204800000 [10659.867355] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10660.266068] Lustre: 295783:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13753-MDT0001-mdtlov: can't create new llog: rc = -115 [10660.281189] LustreError: 295783:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13753-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10664.791666] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10666.485295] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10669.787717] Lustre: server umount fs13753-MDT0001 complete [10679.264889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10679.271058] LustreError: Skipped 1 previous similar message [10679.275340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10679.287604] Lustre: Skipped 3 previous similar messages [10679.292679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10684.336947] Lustre: server umount lustre-MDT0000 complete [10691.407913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10691.506788] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10695.818046] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10697.306373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10703.505278] loop1: detected capacity change from 0 to 204800000 [10708.552963] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10708.818834] Lustre: 298227:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13753-MDT0001-mdtlov: can't create new llog: rc = -115 [10708.829804] LustreError: 298227:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13753-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10713.176490] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10717.526985] Lustre: server umount fs13753-MDT0001 complete [10740.704240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10745.962768] Lustre: server umount lustre-OST0000 complete [10753.512044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10753.519155] Lustre: Skipped 9 previous similar messages [10755.580550] Lustre: server umount lustre-MDT0000 complete [10759.352322] LustreError: 289997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424609 with bad export cookie 1783876185256965053 [10759.363099] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10759.371625] LustreError: 289997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10759.700458] Lustre: server umount lustre-MDT0001 complete [10769.435982] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [10772.341779] Key type lgssc unregistered [10772.543790] LNet: 300571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10772.548874] LNetError: 300571:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10772.561903] LNet: Removed LNI 192.168.203.117@tcp [10773.340181] Key type .llcrypt unregistered [10773.344761] Key type ._llcrypt unregistered [10792.184272] Key type ._llcrypt registered [10792.186415] Key type .llcrypt registered [10792.299711] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [10793.282773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10793.333251] alg: No test for adler32 (adler32-zlib) [10794.918916] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [10795.193482] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [10796.863667] Key type lgssc registered [10797.831818] Lustre: Echo OBD driver; http://www.lustre.org/ [10805.506369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10805.522667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10807.092708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10810.752996] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10816.483938] LustreError: 301840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10816.500215] LustreError: 301840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10818.792318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10819.090063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10823.931121] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10833.175676] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10838.741269] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10846.248766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10846.607402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10851.752767] LustreError: 303670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10852.437485] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10856.935802] LustreError: 303671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10859.620560] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10861.486383] LustreError: 303978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10866.549645] LustreError: 303668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10866.558821] LustreError: 303668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10876.834951] LustreError: 303668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10876.850806] LustreError: 303668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10880.600535] loop1: detected capacity change from 0 to 204800000 [10880.791293] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10880.814604] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10880.861407] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10885.991445] loop1: detected capacity change from 0 to 204800000 [10891.987064] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10892.204735] Lustre: fs20177-OST0000: new disk, initializing [10892.211107] Lustre: srv-fs20177-OST0000: No data found on store. Initialize space. [10892.234757] Lustre: 305576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10892.328397] Lustre: fs20177-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10893.817351] LustreError: 303668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10893.854416] LustreError: 303668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [10898.787747] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10906.311691] Lustre: server umount fs20177-OST0000 complete [10919.349987] loop1: detected capacity change from 0 to 204800000 [10925.014881] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10925.181488] Lustre: fs20177-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10927.978983] LustreError: 303670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10928.026913] LustreError: 303670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [10929.637801] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10930.652888] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10938.677679] loop2: detected capacity change from 0 to 204800000 [10938.882133] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10938.891373] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10938.909962] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10943.067963] loop2: detected capacity change from 0 to 204800000 [10945.001646] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10948.041522] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10948.094685] LustreError: fs20177-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10948.102313] LustreError: fs20177-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10948.108685] LustreError: 308616:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10948.112988] LustreError: 308616:0:(tgt_mount.c:2129:server_put_super()) no obd fs20177-OST0001 [10948.116406] LustreError: 308616:0:(tgt_mount.c:116:server_deregister_mount()) fs20177-OST0001 not registered [10948.157979] Lustre: server umount fs20177-OST0001 complete [10948.160580] LustreError: 308616:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10957.427690] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10957.524393] Lustre: fs20177-OST0001: new disk, initializing [10957.527797] Lustre: srv-fs20177-OST0001: No data found on store. Initialize space. [10957.533213] Lustre: 309168:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10957.585726] Lustre: fs20177-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10962.564942] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [10969.667947] Lustre: server umount fs20177-OST0000 complete [10977.871494] Lustre: server umount fs20177-OST0001 complete [10988.685379] loop1: detected capacity change from 0 to 204800000 [10994.155289] LustreError: 303670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10994.184224] LustreError: 303670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 39 previous similar messages [10995.066074] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10995.200236] Lustre: fs20177-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11001.414875] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11008.633802] loop2: detected capacity change from 0 to 204800000 [11014.645787] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11021.713855] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11030.685802] Lustre: server umount fs20177-OST0000 complete [11038.501407] Lustre: server umount fs20177-OST0001 complete [11050.466289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11050.480085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11050.512657] Lustre: Skipped 1 previous similar message [11050.531962] Lustre: Skipped 1 previous similar message [11055.588195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11055.879966] Lustre: server umount lustre-OST0000 complete [11059.685882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11059.692784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11059.710144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11059.718557] Lustre: Skipped 1 previous similar message [11060.715133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11065.710372] Lustre: server umount lustre-MDT0000 complete [11069.367522] LustreError: 301820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424919 with bad export cookie 10445607114495574354 [11069.429219] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11069.431320] LustreError: 301820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11069.993520] Lustre: server umount lustre-MDT0001 complete [11080.297248] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [11082.646300] Key type lgssc unregistered [11082.846765] LNet: 315090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11082.851954] LNetError: 315090:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11082.863705] LNet: Removed LNI 192.168.203.117@tcp [11083.992257] Key type .llcrypt unregistered [11084.000336] Key type ._llcrypt unregistered [11116.173920] Key type ._llcrypt registered [11116.181739] Key type .llcrypt registered [11116.419807] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [11132.125513] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [11133.169205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11133.183144] alg: No test for adler32 (adler32-zlib) [11134.956812] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [11135.232440] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [11136.991246] Key type lgssc registered [11138.583697] Lustre: Echo OBD driver; http://www.lustre.org/ [11173.808670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11173.890348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11175.106421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11175.161551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11175.239617] Lustre: lustre-MDT0000: new disk, initializing [11175.395595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11175.505776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11180.069940] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11190.072830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11190.219486] Lustre: 318956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11190.276493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11190.280565] Lustre: Skipped 1 previous similar message [11190.353522] Lustre: lustre-MDT0001: new disk, initializing [11190.500976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11190.546062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11190.569747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11195.127852] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11204.806055] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11210.989523] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11217.087935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11217.435917] Lustre: lustre-OST0000: new disk, initializing [11217.447234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11217.454454] Lustre: 320920:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11217.573195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11223.755716] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11224.628051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11224.637573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11224.757554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11232.812325] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11234.507420] LustreError: 317126:0:(lmv_obd.c:208:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [11234.551995] Lustre: Mounted lustre-client [11235.702953] Lustre: Unmounted lustre-client [11245.025867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11245.035046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11245.038177] Lustre: Skipped 1 previous similar message [11249.863703] Lustre: server umount lustre-OST0000 complete [11255.265086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11255.266265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11255.287855] Lustre: Skipped 2 previous similar messages [11260.249900] Lustre: server umount lustre-MDT0000 complete [11264.734991] LustreError: 322330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425114 with bad export cookie 5395655118987089401 [11264.737200] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11264.745588] LustreError: 322330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11265.076356] Lustre: server umount lustre-MDT0001 complete [11278.069372] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [11279.875330] Key type lgssc unregistered [11292.402338] LNet: 323083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11292.406753] LNetError: 323083:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11363.431739] LNet: Removed LNI 192.168.203.117@tcp [11364.337419] Key type .llcrypt unregistered [11364.340659] Key type ._llcrypt unregistered [11384.128682] Key type ._llcrypt registered [11384.145916] Key type .llcrypt registered [11384.230509] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [11399.814744] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [11400.916373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11401.049461] alg: No test for adler32 (adler32-zlib) [11402.682193] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [11402.951179] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [11404.583158] Key type lgssc registered [11405.790922] Lustre: Echo OBD driver; http://www.lustre.org/ [11432.263365] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:14:40 (1786425280) [11438.526101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11438.541179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11440.200147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11440.229079] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11440.313365] Lustre: lustre-MDT0000: new disk, initializing [11440.385992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11440.408149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11445.195939] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11455.371600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11455.467896] Lustre: 326304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11455.538346] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11455.540921] Lustre: Skipped 1 previous similar message [11455.588590] Lustre: lustre-MDT0001: new disk, initializing [11455.662348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11455.694765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11455.716082] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11459.792549] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11468.908629] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11474.283844] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11481.001182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11481.233756] Lustre: lustre-OST0000: new disk, initializing [11481.236991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11481.242602] Lustre: 328271:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11481.317185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11487.324442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11487.337563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11487.422617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11487.763947] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11496.785400] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11513.312748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11513.312784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11513.321694] LustreError: Skipped 1 previous similar message [11513.332168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11518.014418] Lustre: server umount lustre-OST0000 complete [11522.543472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11522.547725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11522.565382] Lustre: Skipped 1 previous similar message [11522.576332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11522.588124] Lustre: Skipped 1 previous similar message [11523.555325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11523.577271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11523.585681] Lustre: Skipped 1 previous similar message [11528.678741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11528.692494] Lustre: Skipped 1 previous similar message [11533.795987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11533.806050] Lustre: Skipped 1 previous similar message [11535.327141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11535.533536] Lustre: server umount lustre-MDT0000 complete [11539.692341] LustreError: 328992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425389 with bad export cookie 7190863479717750983 [11539.697899] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11539.708258] LustreError: 328992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11540.204291] Lustre: server umount lustre-MDT0001 complete [11549.948189] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [11552.261433] Key type lgssc unregistered [11552.456843] LNet: 330180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11552.462443] LNetError: 330180:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11552.476641] LNet: Removed LNI 192.168.203.117@tcp [11553.182531] Key type .llcrypt unregistered [11553.190223] Key type ._llcrypt unregistered [11576.422990] Key type ._llcrypt registered [11576.425395] Key type .llcrypt registered [11576.566462] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [11578.259631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11578.535181] alg: No test for adler32 (adler32-zlib) [11579.532905] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [11579.800960] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [11581.511200] Key type lgssc registered [11582.298332] Lustre: Echo OBD driver; http://www.lustre.org/ [11589.343964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11589.368888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11590.645068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11595.057768] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11600.872686] LustreError: 331457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11600.895129] LustreError: 331457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11601.878569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11602.094570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11607.140460] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11614.672334] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11620.839675] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11628.087625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11628.291962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11634.015813] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11640.826431] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11654.642266] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11656.855871] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11662.310611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11662.325816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11662.328085] Lustre: Skipped 1 previous similar message [11662.392527] Lustre: Skipped 2 previous similar messages [11664.277625] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.17@tcp (stopping) [11667.429197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11667.438473] Lustre: Skipped 2 previous similar messages [11667.997543] Lustre: server umount lustre-MDT0000 complete [11669.341993] LustreError: 331453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11671.275461] LustreError: 331437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425521 with bad export cookie 16400199006110605702 [11671.277127] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11671.282863] LustreError: 331437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11672.544294] LustreError: 331454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11672.548852] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11672.566806] Lustre: Skipped 1 previous similar message [11672.571570] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11676.576313] LustreError: 331452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11676.578740] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11676.594049] LustreError: 331452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11676.607250] Lustre: Skipped 1 previous similar message [11677.652055] Lustre: server umount lustre-MDT0001 complete [11681.072502] Lustre: server umount lustre-OST0000 complete [11689.273143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11693.476407] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11695.752478] Lustre: 335637:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.117@tcp [11697.845825] Lustre: server umount MGS complete [11706.675292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11706.971422] LustreError: 336208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11707.033582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11711.117540] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11716.577429] LustreError: 336208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11716.599019] LustreError: 336208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11718.718969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11718.909691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11722.830746] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11730.688201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11730.974040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11736.944458] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11743.210527] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11744.885645] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11750.759192] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11752.723477] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11757.543922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11757.548694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11757.556650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11758.566745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11762.918275] Lustre: server umount lustre-OST0000 complete [11768.800662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11768.836556] Lustre: Skipped 2 previous similar messages [11772.620614] Lustre: server umount lustre-MDT0000 complete [11773.921706] LustreError: 338263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11773.937297] LustreError: 338263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11775.931599] LustreError: 336943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425625 with bad export cookie 16400199006110606542 [11775.934871] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11775.948522] LustreError: 336943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11776.308856] Lustre: server umount lustre-MDT0001 complete [11787.640224] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [11790.064587] Key type lgssc unregistered [11790.395846] LNet: 339642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11790.399636] LNetError: 339642:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11790.407849] LNet: Removed LNI 192.168.203.117@tcp [11791.213662] Key type .llcrypt unregistered [11791.221140] Key type ._llcrypt unregistered [11810.462059] Key type ._llcrypt registered [11810.474725] Key type .llcrypt registered [11810.596619] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [11827.370818] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [11829.637552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11829.802428] alg: No test for adler32 (adler32-zlib) [11830.761657] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [11830.967110] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [11832.639175] Key type lgssc registered [11833.650389] Lustre: Echo OBD driver; http://www.lustre.org/ [11872.420832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11872.454424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11873.729438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11873.747120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11873.795699] Lustre: lustre-MDT0000: new disk, initializing [11873.853079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11873.863760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11877.684407] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11888.579316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11888.805820] Lustre: 342807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11889.028908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11889.032353] Lustre: Skipped 1 previous similar message [11889.084531] Lustre: lustre-MDT0001: new disk, initializing [11889.180261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11889.299325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11889.331097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11894.411349] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11902.716588] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11909.297259] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11916.544767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11917.332499] Lustre: lustre-OST0000: new disk, initializing [11917.338982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11917.353407] Lustre: 344776:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11917.527282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11925.517942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11925.526179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11925.827889] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [11925.839604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11935.531650] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11940.906803] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11941.138453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11945.137967] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11945.382208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11951.075477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11951.081888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11951.106351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11951.588499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11951.603371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11951.615570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11956.705562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11956.726576] Lustre: Skipped 1 previous similar message [11961.311132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11961.423517] Lustre: server umount lustre-OST0000 complete [11965.930607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11965.940568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11965.958934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11972.064107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11972.072881] Lustre: Skipped 3 previous similar messages [11979.272775] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11979.730908] Lustre: server umount lustre-MDT0000 complete [11984.169464] LustreError: 342797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425834 with bad export cookie 3821639448770858783 [11984.182491] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11984.187601] LustreError: 342797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11984.422629] Lustre: server umount lustre-MDT0001 complete [11994.248341] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:24:02 (1786425842) [12009.377992] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [12018.323917] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [12050.860715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12051.123267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12051.152448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12051.260374] Lustre: lustre-MDT0000: new disk, initializing [12051.334234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12051.370253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12056.402647] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [12066.862335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12066.982927] Lustre: 349532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12067.024431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12067.027799] Lustre: Skipped 1 previous similar message [12067.080928] Lustre: lustre-MDT0001: new disk, initializing [12067.158989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12067.211163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12067.224488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12070.710796] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [12079.461958] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12084.946712] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12092.748260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12093.016618] Lustre: lustre-OST0000: new disk, initializing [12093.019741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12093.025804] Lustre: 351500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12093.133257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12099.129786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12099.144107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12099.649159] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [12099.703876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12110.665621] Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12142.774319] Lustre: server umount lustre-OST0000 complete [12149.217824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12149.232677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12149.253955] Lustre: Skipped 1 previous similar message [12149.262948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12149.270892] Lustre: Skipped 2 previous similar messages [12157.016955] Lustre: server umount lustre-MDT0000 complete [12160.563836] LustreError: 349526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786426010 with bad export cookie 3821639448770859665 [12160.587597] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12160.590882] LustreError: 349526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12171.464415] Lustre: server umount lustre-MDT0001 complete [12183.740651] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12186.246279] Key type lgssc unregistered [12186.450737] LNet: 353654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12186.460677] LNetError: 353654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12186.477850] LNet: Removed LNI 192.168.203.117@tcp [12187.570824] Key type .llcrypt unregistered [12187.572911] Key type ._llcrypt unregistered [12279.633805] Key type ._llcrypt registered [12279.635867] Key type .llcrypt registered [12279.734417] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12280.125773] Key type .llcrypt unregistered [12280.132523] Key type ._llcrypt unregistered [12360.908354] Key type ._llcrypt registered [12360.910552] Key type .llcrypt registered [12361.005658] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12361.326596] Key type .llcrypt unregistered [12361.329722] Key type ._llcrypt unregistered [12394.669939] Key type ._llcrypt registered [12394.680522] Key type .llcrypt registered [12394.812596] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12395.084586] Key type .llcrypt unregistered [12395.087196] Key type ._llcrypt unregistered [12426.917517] Key type ._llcrypt registered [12426.920598] Key type .llcrypt registered [12427.122587] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12427.393623] Key type .llcrypt unregistered [12427.395728] Key type ._llcrypt unregistered [12501.092537] Key type ._llcrypt registered [12501.094624] Key type .llcrypt registered [12501.333781] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12501.602990] Key type .llcrypt unregistered [12501.606429] Key type ._llcrypt unregistered [12530.243376] Key type ._llcrypt registered [12530.245782] Key type .llcrypt registered [12530.332567] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12530.533310] Key type .llcrypt unregistered [12530.535895] Key type ._llcrypt unregistered [12561.748336] Key type ._llcrypt registered [12561.750451] Key type .llcrypt registered [12561.832133] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12562.043377] Key type .llcrypt unregistered [12562.047329] Key type ._llcrypt unregistered [12590.303174] Key type ._llcrypt registered [12590.305252] Key type .llcrypt registered [12590.397684] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12590.965617] Key type .llcrypt unregistered [12590.968547] Key type ._llcrypt unregistered [12620.202646] Key type ._llcrypt registered [12620.204704] Key type .llcrypt registered [12620.344952] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12620.636327] Key type .llcrypt unregistered [12620.638380] Key type ._llcrypt unregistered [12650.009382] Key type ._llcrypt registered [12650.010853] Key type .llcrypt registered [12650.083977] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12650.387383] Key type .llcrypt unregistered [12650.389118] Key type ._llcrypt unregistered [12714.851498] Key type ._llcrypt registered [12714.855494] Key type .llcrypt registered [12714.955651] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12715.249363] Key type .llcrypt unregistered [12715.252493] Key type ._llcrypt unregistered [12747.013391] Key type ._llcrypt registered [12747.016367] Key type .llcrypt registered [12747.151384] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12747.490505] Key type .llcrypt unregistered [12747.493499] Key type ._llcrypt unregistered [12783.943841] Key type ._llcrypt registered [12783.946383] Key type .llcrypt registered [12784.011365] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [12784.193885] Key type .llcrypt unregistered [12784.196604] Key type ._llcrypt unregistered [12837.212295] Key type ._llcrypt registered [12837.222587] Key type .llcrypt registered [12837.368911] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [12852.954532] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [12853.963810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12854.040119] alg: No test for adler32 (adler32-zlib) [12855.020922] Lustre: Lustre: Build Version: 2.17.54_107_gf473267 [12855.215242] LNet: Added LNI 192.168.203.117@tcp [8/256/0/180] [12856.847222] Key type lgssc registered [12857.924774] Lustre: Echo OBD driver; http://www.lustre.org/ [12891.705074] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12891.727335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12893.016129] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12893.037349] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12893.096699] Lustre: tst250fs-MDT0000: new disk, initializing [12893.152362] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12893.164873] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12897.174736] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [12909.132668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12909.281994] Lustre: 373934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12909.344599] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12909.348268] Lustre: Skipped 1 previous similar message [12909.405474] Lustre: tst250fs-MDT0001: new disk, initializing [12909.487620] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12909.537805] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12909.544563] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12913.789437] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [12919.063216] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12926.307994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12926.653456] Lustre: tst250fs-OST0000: new disk, initializing [12926.657508] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12926.664631] Lustre: 375567:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12926.910695] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12927.667248] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12927.677711] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12927.960333] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12933.519382] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [12946.392445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12946.883339] Lustre: tst250fs-OST0001: new disk, initializing [12946.893067] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12946.916548] Lustre: 376440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12947.062808] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12953.712488] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug -1 all [12955.299968] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12955.310142] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12955.484276] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12959.248476] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12961.465846] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [12961.471467] Lustre: Skipped 1 previous similar message [12981.221267] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12981.221810] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12981.230094] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12981.241554] Lustre: Skipped 2 previous similar messages [12986.337923] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12986.349489] Lustre: Skipped 6 previous similar messages [12986.656881] LustreError: 377767:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x7366303532747374:0x0:0x0].0x0 (ffff8b4341147e00) refcount nonzero (2) after lock cleanup; forcing cleanup. [12986.725932] Lustre: server umount tst250fs-MDT0000 complete [12989.925596] LustreError: 375566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786426839 with bad export cookie 15631799201035851384 [12989.933529] LustreError: MGC192.168.203.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12989.937340] LustreError: 375566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12990.305878] Lustre: server umount tst250fs-MDT0001 complete [13004.113522] Lustre: server umount tst250fs-OST0000 complete [13017.950585] Lustre: server umount tst250fs-OST0001 complete [13034.318957] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [13041.559653] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [13072.195567] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:42:00 (1786426920) [13074.281309] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13090.698893] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_hostid [13101.495366] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing load_modules_local [13129.767734] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12814 sec ========= 01:42:57 (1786426977) [13131.307281] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:42:59 (1786426979) === [13135.150804] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:43:03 (1786426983) === [13173.910689] Lustre: DEBUG MARKER: oleg317-server.virtnet: executing unload_modules_local [13176.533986] Key type lgssc unregistered [13176.784507] LNet: 385582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13176.791476] LNetError: 385582:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13176.815960] LNet: Removed LNI 192.168.203.117@tcp [13177.581177] Key type .llcrypt unregistered [13177.586438] Key type ._llcrypt unregistered