[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 519499399 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002263] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006879] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009136] LSM: Security Framework initializing [ 0.011029] Yama: becoming mindful. [ 0.011592] SELinux: Initializing. [ 0.012043] *** VALIDATE selinux *** [ 0.019230] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023242] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025080] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027011] *** VALIDATE tmpfs *** [ 0.028476] *** VALIDATE proc *** [ 0.029361] *** VALIDATE cgroup *** [ 0.030006] *** VALIDATE cgroup2 *** [ 0.032069] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033120] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034022] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035026] Spectre V2 : User space: Vulnerable [ 0.036005] Speculative Store Bypass: Vulnerable [ 0.039272] debug: unmapping init [mem 0xffffffffb2259000-0xffffffffb2260fff] [ 0.042000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042633] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043024] ... version: 2 [ 0.044013] ... bit width: 48 [ 0.045014] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047015] ... max period: 00007fffffffffff [ 0.048013] ... fixed-purpose events: 3 [ 0.048957] ... event mask: 000000070000000f [ 0.049254] rcu: Hierarchical SRCU implementation. [ 0.051122] smp: Bringing up secondary CPUs ... [ 0.052599] x86: Booting SMP configuration: [ 0.053029] .... node #0, CPUs: #1 #2 #3 [ 0.056138] smp: Brought up 1 node, 4 CPUs [ 0.058010] smpboot: Max logical packages: 1 [ 0.058937] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.107000] node 0 deferred pages initialised in 48ms [ 0.111006] devtmpfs: initialized [ 0.111904] x86/mm: Memory block size: 128MB [ 0.113144] gcov: version magic: 0x41383552 [ 0.115199] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.116083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.117331] pinctrl core: initialized pinctrl subsystem [ 0.118226] [ 0.118889] ************************************************************* [ 0.119012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.120018] ** ** [ 0.121015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.122016] ** ** [ 0.123011] ** This means that this kernel is built to expose internal ** [ 0.124115] ** IOMMU data structures, which may compromise security on ** [ 0.125015] ** your system. ** [ 0.126015] ** ** [ 0.127017] ** If you see this message and you are not debugging the ** [ 0.128014] ** kernel, report this immediately to your vendor! ** [ 0.129015] ** ** [ 0.130014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131027] ************************************************************* [ 0.132633] NET: Registered protocol family 16 [ 0.133476] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.134062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.135136] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.136478] cpuidle: using governor menu [ 0.137840] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.141601] PCI: Using configuration type 1 for base access [ 0.143120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.152167] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.153031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.154132] cryptd: max_cpu_qlen set to 1000 [ 0.156092] ACPI: Added _OSI(Module Device) [ 0.157010] ACPI: Added _OSI(Processor Device) [ 0.158011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.159009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.162976] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.164576] ACPI: Interpreter enabled [ 0.165057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.166011] ACPI: Using IOAPIC for interrupt routing [ 0.167101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.168370] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.177376] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.178050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.179014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.180082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.182353] acpiphp: Slot [2] registered [ 0.183128] acpiphp: Slot [5] registered [ 0.184121] acpiphp: Slot [6] registered [ 0.185114] acpiphp: Slot [7] registered [ 0.186105] acpiphp: Slot [8] registered [ 0.187111] acpiphp: Slot [9] registered [ 0.188151] acpiphp: Slot [10] registered [ 0.189135] acpiphp: Slot [3] registered [ 0.190151] acpiphp: Slot [4] registered [ 0.192117] acpiphp: Slot [11] registered [ 0.194271] acpiphp: Slot [12] registered [ 0.197122] acpiphp: Slot [13] registered [ 0.199126] acpiphp: Slot [14] registered [ 0.201111] acpiphp: Slot [15] registered [ 0.204152] acpiphp: Slot [16] registered [ 0.206113] acpiphp: Slot [17] registered [ 0.209213] acpiphp: Slot [18] registered [ 0.211176] acpiphp: Slot [19] registered [ 0.213107] acpiphp: Slot [20] registered [ 0.215147] acpiphp: Slot [21] registered [ 0.218141] acpiphp: Slot [22] registered [ 0.220140] acpiphp: Slot [23] registered [ 0.223100] acpiphp: Slot [24] registered [ 0.224095] acpiphp: Slot [25] registered [ 0.226106] acpiphp: Slot [26] registered [ 0.227173] acpiphp: Slot [27] registered [ 0.229173] acpiphp: Slot [28] registered [ 0.230105] acpiphp: Slot [29] registered [ 0.232111] acpiphp: Slot [30] registered [ 0.233109] acpiphp: Slot [31] registered [ 0.235085] PCI host bridge to bus 0000:00 [ 0.236021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.238023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.240021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.243023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.245022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.248025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.250173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.253040] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.256343] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.267693] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.272599] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.275021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.278017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.280018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.283695] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.286704] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.289042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.291797] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.297015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.310018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.316915] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.322035] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.327017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.334015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.344015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.352912] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.358016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.362013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.375021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.385000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.399013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.407018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.422019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.430825] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.438017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.444019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.461023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.469078] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.479016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.488033] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.509026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.520203] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.527018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.543019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.573019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.588861] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.592477] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.594671] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.600062] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.603313] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.608050] iommu: Default domain type: Passthrough [ 0.610674] SCSI subsystem initialized [ 0.612222] ACPI: bus type USB registered [ 0.614173] usbcore: registered new interface driver usbfs [ 0.616142] usbcore: registered new interface driver hub [ 0.617105] usbcore: registered new device driver usb [ 0.619231] pps_core: LinuxPPS API ver. 1 registered [ 0.621021] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.624107] PTP clock support registered [ 0.627084] EDAC MC: Ver: 3.0.0 [ 0.628547] PCI: Using ACPI for IRQ routing [ 0.630926] NetLabel: Initializing [ 0.632012] NetLabel: domain hash size = 128 [ 0.634011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.636081] NetLabel: unlabeled traffic allowed by default [ 0.638163] vgaarb: loaded [ 0.640422] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.642022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.650000] clocksource: Switched to clocksource kvm-clock [ 0.754880] VFS: Disk quotas dquot_6.6.0 [ 0.756552] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.759114] *** VALIDATE ramfs *** [ 0.760428] *** VALIDATE hugetlbfs *** [ 0.761978] pnp: PnP ACPI init [ 0.764414] pnp: PnP ACPI: found 6 devices [ 0.781729] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.784628] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.786707] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.788749] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.791099] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.793836] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.797080] NET: Registered protocol family 2 [ 0.799612] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.804560] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.808341] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.814093] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.817557] TCP: Hash tables configured (established 65536 bind 65536) [ 0.820592] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.823935] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.826860] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.830060] NET: Registered protocol family 1 [ 0.833231] RPC: Registered named UNIX socket transport module. [ 0.835306] RPC: Registered udp transport module. [ 0.837155] RPC: Registered tcp transport module. [ 0.838912] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.841656] NET: Registered protocol family 44 [ 0.843478] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.845645] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.847903] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.850247] PCI: CLS 0 bytes, default 64 [ 0.851723] Unpacking initramfs... [ 2.315227] debug: unmapping init [mem 0xffff8f0dfcc54000-0xffff8f0dfffbffff] [ 2.319425] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.321816] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.324750] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.838825] Initialise system trusted keyrings [ 2.841282] Key type blacklist registered [ 2.844126] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.854468] zbud: loaded [ 2.859102] *** VALIDATE nfs *** [ 2.860841] *** VALIDATE nfs4 *** [ 2.862934] pstore: using deflate compression [ 2.867190] Platform Keyring initialized [ 2.967746] NET: Registered protocol family 38 [ 2.970281] Key type asymmetric registered [ 2.972496] Asymmetric key parser 'x509' registered [ 2.975197] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.979452] io scheduler mq-deadline registered [ 2.982037] io scheduler kyber registered [ 2.984393] io scheduler bfq registered [ 2.986741] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.991035] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.995285] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.999334] ACPI: Power Button [PWRF] [ 3.006125] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.012899] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.023902] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.029770] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.041970] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.067718] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.094323] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.099201] Non-volatile memory driver v1.3 [ 3.101688] Linux agpgart interface v0.103 [ 3.132717] virtio_blk virtio1: [vda] 149424 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.136882] vda: detected capacity change from 0 to 76505088 [ 3.156791] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.161312] vdb: detected capacity change from 0 to 1073741824 [ 3.177814] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.180355] vdc: detected capacity change from 0 to 2621440000 [ 3.204071] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.207019] vdd: detected capacity change from 0 to 2621440000 [ 3.226553] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.230849] vde: detected capacity change from 0 to 4294967296 [ 3.246809] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.251303] vdf: detected capacity change from 0 to 4294967296 [ 3.259484] libphy: Fixed MDIO Bus: probed [ 3.268201] usbcore: registered new interface driver usbserial_generic [ 3.270406] usbserial: USB Serial support registered for generic [ 3.273072] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.277812] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.280037] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.282891] mousedev: PS/2 mouse device common for all mice [ 3.285237] rtc_cmos 00:05: RTC can wake from S4 [ 3.288217] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.288395] rtc_cmos 00:05: registered as rtc0 [ 3.295116] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.297747] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.300213] intel_pstate: CPU model not supported [ 3.302484] hid: raw HID events driver (C) Jiri Kosina [ 3.307841] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.308458] usbcore: registered new interface driver usbhid [ 3.314854] usbhid: USB HID core driver [ 3.316229] drop_monitor: Initializing network drop monitor service [ 3.318173] Initializing XFRM netlink socket [ 3.319961] NET: Registered protocol family 10 [ 3.323711] Segment Routing with IPv6 [ 3.325056] NET: Registered protocol family 17 [ 3.326745] mpls_gso: MPLS GSO support [ 3.333838] RAS: Correctable Errors collector initialized. [ 3.336851] AVX version of gcm_enc/dec engaged. [ 3.339395] AES CTR mode by8 optimization enabled [ 3.423758] sched_clock: Marking stable (3423723700, 0)->(4354103660, -930379960) [ 3.426573] registered taskstats version 1 [ 3.428246] Loading compiled-in X.509 certificates [ 3.429779] zswap: loaded using pool lzo/zbud [ 3.454531] Key type big_key registered [ 3.467588] Key type encrypted registered [ 3.469445] ima: No TPM chip found, activating TPM-bypass! [ 3.471895] ima: Allocated hash algorithm: sha1 [ 3.474061] ima: No architecture policies found [ 3.476324] evm: Initialising EVM extended attributes: [ 3.478302] evm: security.selinux [ 3.479741] evm: security.ima [ 3.481116] evm: security.capability [ 3.482588] evm: HMAC attrs: 0x1 [ 3.485234] rtc_cmos 00:05: setting system clock to 2026-08-22 08:28:02 UTC (1787387282) [ 3.492294] debug: unmapping init [mem 0xffffffffb3203000-0xffffffffb33fffff] [ 3.495612] debug: unmapping init [mem 0xffffffffb1f82000-0xffffffffb2258fff] [ 3.505119] Write protecting the kernel read-only data: 28672k [ 3.508821] debug: unmapping init [mem 0xffffffffb0603000-0xffffffffb07fffff] [ 3.511977] debug: unmapping init [mem 0xffffffffb0f14000-0xffffffffb0ffffff] [ 3.549240] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.556250] systemd[1]: Detected virtualization kvm. [ 3.557718] systemd[1]: Detected architecture x86-64. [ 3.559237] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.590863] systemd[1]: No hostname configured. [ 3.592331] systemd[1]: Set hostname to . [ 3.594107] random: systemd: uninitialized urandom read (16 bytes read) [ 3.596116] systemd[1]: Initializing machine ID from random generator. [ 3.662205] random: ln: uninitialized urandom read (6 bytes read) [ 3.752539] random: systemd: uninitialized urandom read (16 bytes read) [ 3.754938] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.758970] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.762219] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.410633] device-mapper: uevent: version 1.0.3 [ 4.413173] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.165153] random: fast init done [ 5.171517] virtio_net virtio0 ens2: renamed from eth0 [ 5.215304] scsi host0: ata_piix [ 5.230233] scsi host1: ata_piix [ 5.233573] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.236293] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.592121] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.988356] random: crng init done [ 9.989833] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.447425] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.576304] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.804111] SELinux: Disabled at runtime. [ 11.863418] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.872471] systemd[1]: Detected virtualization kvm. [ 11.874636] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.500710] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.503243] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.506991] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.510310] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.514318] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.522458] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.529806] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.555981] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.025528] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.368730] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.424944] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.558414] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.582099] EDAC sbridge: Ver: 1.1.2 [ 15.295852] Key type dns_resolver registered [ 15.605907] NFS: Registering the id_resolver key type [ 15.607911] Key type id_resolver registered [ 15.609623] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg105-server login: [ 41.823516] libcfs: loading out-of-tree module taints kernel. [ 41.845247] Key type ._llcrypt registered [ 41.846831] Key type .llcrypt registered [ 41.901095] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 58.451423] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 61.015811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.030149] alg: No test for adler32 (adler32-zlib) [ 62.671844] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 64.204479] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 66.055240] Key type lgssc registered [ 68.233842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.450087] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 97.546468] hrtimer: interrupt took 3450823 ns [ 138.702125] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 156.291182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 156.303681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.959757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 158.129182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 158.304851] Lustre: lustre-MDT0000: new disk, initializing [ 158.483498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.536858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 164.483847] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.432898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.534989] Lustre: 6527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.558339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.561848] Lustre: Skipped 1 previous similar message [ 179.631355] Lustre: lustre-MDT0001: new disk, initializing [ 179.677914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.702978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.716020] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.995326] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.115545] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.779964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.032686] Lustre: lustre-OST0000: new disk, initializing [ 200.035904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 200.041992] Lustre: 8464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 200.129575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.192086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 203.215677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 203.265785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 207.179223] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.985588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.136811] Lustre: lustre-OST0001: new disk, initializing [ 222.142903] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 222.148740] Lustre: 9538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.251372] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.984196] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.969479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 230.978580] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.081451] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.091836] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.107428] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 259.079470] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.860672] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing yml_node [ 272.809464] Lustre: DEBUG MARKER: Client: 2.17.54.111 [ 275.772716] Lustre: DEBUG MARKER: MDS: 2.17.54.111 [ 278.604559] Lustre: DEBUG MARKER: OSS: 2.17.54.111 [ 280.565659] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 04:32:39 EDT 2026 [ 299.880516] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 302.229335] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 323.039887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 323.042077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.046311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.080801] Lustre: Skipped 2 previous similar messages [ 328.161227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.164112] Lustre: Skipped 3 previous similar messages [ 333.281802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.284532] Lustre: Skipped 1 previous similar message [ 333.292474] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 333.830782] Lustre: server umount lustre-MDT0000 complete [ 339.626957] LustreError: 6520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387618 with bad export cookie 13872460823805202957 [ 339.634468] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 339.651562] LustreError: 6520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 343.525129] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.526087] LustreError: 6536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.557826] Lustre: Skipped 1 previous similar message [ 343.589401] LustreError: 6536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 343.596359] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.598436] Lustre: Skipped 1 previous similar message [ 344.545678] LustreError: 8883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.560218] LustreError: 8883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 346.310534] Lustre: server umount lustre-MDT0001 complete [ 351.096319] Lustre: server umount lustre-OST0000 complete [ 355.240439] Lustre: server umount lustre-OST0001 complete [ 362.579499] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 373.802822] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 418.603224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.918488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 418.977366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 419.158586] Lustre: lustre-MDT0000: new disk, initializing [ 419.253513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.307468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 424.091734] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 435.235224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.376835] Lustre: 15097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 435.420971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 435.424840] Lustre: Skipped 1 previous similar message [ 435.504768] Lustre: lustre-MDT0001: new disk, initializing [ 435.607650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 435.635828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 435.648441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 440.430607] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 450.857024] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 456.581397] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 463.908421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 464.240620] Lustre: lustre-OST0000: new disk, initializing [ 464.253538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 464.265475] Lustre: 17066:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 464.372229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.672854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 465.689413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 465.919489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 471.124935] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 481.961323] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 487.133922] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 487.472815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 493.051391] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 493.359237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 496.611217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.613443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.631876] LustreError: Skipped 1 previous similar message [ 496.640379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.659265] Lustre: Skipped 2 previous similar messages [ 496.675158] Lustre: Skipped 3 previous similar messages [ 502.016636] Lustre: server umount lustre-OST0000 complete [ 506.848882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.853955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 506.857416] Lustre: Skipped 3 previous similar messages [ 512.592895] Lustre: server umount lustre-MDT0000 complete [ 516.963630] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387795 with bad export cookie 13872460823805204490 [ 516.969417] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.972077] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 517.346487] Lustre: server umount lustre-MDT0001 complete [ 523.370216] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:36:42 (1787387802) [ 545.520753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.167893] LustreError: 19942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.264447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.212526] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 551.395203] LustreError: 19943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.527443] LustreError: 19942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.732946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.124434] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 578.324497] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 585.103989] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 588.715883] Lustre: Failing over lustre-MDT0000 [ 591.005723] Lustre: server umount lustre-MDT0000 complete [ 591.347761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.368295] Lustre: Skipped 1 previous similar message [ 591.380980] LustreError: 21183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.711411] Lustre: 3641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387870/real 1787387870] req@ffff8f0e45be3480 x1874211470742784/t0(0) o400->MGC192.168.201.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787387886 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 607.741823] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 607.765960] LustreError: 20670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.782138] LustreError: 20670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 612.153758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 617.953092] LustreError: 3637:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8f0e45bdf800 x1874211470745344/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 618.591858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 618.598491] Lustre: Skipped 1 previous similar message [ 618.655731] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 623.584844] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 623.606700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 623.656524] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 625.210126] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 639.106531] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 648.162798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.172984] Lustre: Skipped 1 previous similar message [ 648.176778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 648.178614] Lustre: Skipped 2 previous similar messages [ 651.381919] Lustre: server umount lustre-MDT0000 complete [ 653.287925] LustreError: 20670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.305544] LustreError: 20670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 655.781508] LustreError: 21512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387934 with bad export cookie 13872460823805205904 [ 655.781780] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 655.803262] LustreError: 21512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 656.336729] Lustre: server umount lustre-MDT0001 complete [ 667.291990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.996788] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 689.875375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.626873] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 705.727797] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 713.010429] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 715.651250] Lustre: Failing over lustre-MDT0000 [ 715.744776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 715.764597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.794788] Lustre: Skipped 1 previous similar message [ 715.817127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 716.292812] Lustre: server umount lustre-MDT0000 complete [ 721.888775] LustreError: 24014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 721.909298] LustreError: 24014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 732.127681] Lustre: 3638:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387995/real 1787387995] req@ffff8f0e45bdc000 x1874211470776448/t0(0) o400->MGC192.168.201.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787388011 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 732.176613] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 737.805883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 743.394148] LustreError: 25282:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 743.397565] LustreError: 3637:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8f0d41563480 x1874211470779264/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 744.024302] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 749.036072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 749.037017] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 749.049426] Lustre: Skipped 1 previous similar message [ 749.115878] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 749.368799] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 760.579598] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 766.435367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 766.454021] Lustre: Skipped 2 previous similar messages [ 771.980919] Lustre: server umount lustre-MDT0000 complete [ 776.755301] LustreError: 23266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388055 with bad export cookie 13872460823805206842 [ 776.767404] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 776.776034] LustreError: 23266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 777.226885] Lustre: server umount lustre-MDT0001 complete [ 787.552935] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:41:06 (1787388066) [ 806.757093] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 819.710895] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 871.523387] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 886.304370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.855375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 886.977370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 887.054550] Lustre: lustre-MDT0000: new disk, initializing [ 887.134404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 887.138239] Lustre: Skipped 3 previous similar messages [ 887.155285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 893.665930] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 906.570382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 906.737940] Lustre: 29684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 907.053903] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 913.214917] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 924.079202] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 930.318153] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 938.385433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 938.656162] Lustre: 31657:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 939.929143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 939.975333] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 940.121797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 940.157546] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 946.261787] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 958.022955] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 965.003776] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 991.200088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 991.208471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.225102] LustreError: Skipped 1 previous similar message [ 991.238651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.252786] Lustre: Skipped 1 previous similar message [ 991.285568] Lustre: Skipped 5 previous similar messages [ 994.846977] Lustre: server umount lustre-OST0000 complete [ 1015.264247] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1015.568763] Lustre: server umount lustre-MDT0000 complete [ 1016.809535] LustreError: 29695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1016.839484] LustreError: 29695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 1019.596484] LustreError: 29675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388298 with bad export cookie 13872460823805207248 [ 1019.606721] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1019.616118] LustreError: 29675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1020.102844] Lustre: server umount lustre-MDT0001 complete [ 1032.398387] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1036.042179] Key type lgssc unregistered [ 1036.524166] LNet: 33639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.544808] LNetError: 33639:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.573056] LNet: Removed LNI 192.168.201.105@tcp [ 1038.195343] Key type .llcrypt unregistered [ 1038.197194] Key type ._llcrypt unregistered [ 1065.794543] Key type ._llcrypt registered [ 1065.796984] Key type .llcrypt registered [ 1066.116411] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1067.202516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1067.412769] alg: No test for adler32 (adler32-zlib) [ 1068.598458] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 1068.991414] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1070.799204] Key type lgssc registered [ 1072.289445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1082.140880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1082.202651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.137122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1089.852867] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1094.628052] LustreError: 35057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.648538] LustreError: 35057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1099.746632] LustreError: 35056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1100.102368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.745945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1105.772908] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1116.758428] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1124.031892] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1133.644713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1134.173943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1141.594701] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1145.532570] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1150.292569] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1155.807024] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1169.969868] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 04:47:28 (1787388448) [ 1172.137344] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1174.318892] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1184.216090] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 04:47:42 (1787388462) [ 1195.362747] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 04:47:53 (1787388473) [ 1196.703807] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1206.812704] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1206.817379] Lustre: Skipped 2 previous similar messages [ 1208.893262] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 04:48:07 (1787388487) [ 1211.173759] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1211.178702] Lustre: Skipped 2 previous similar messages [ 1220.785387] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1220.789786] Lustre: Skipped 4 previous similar messages [ 1236.389848] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 04:48:35 (1787388515) [ 1239.516491] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1243.841392] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1253.801789] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 04:48:52 (1787388532) [ 1256.084921] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1256.092247] Lustre: Skipped 6 previous similar messages [ 1268.776764] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 04:49:07 (1787388547) [ 1289.185406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1289.188340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.204400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1289.222141] Lustre: Skipped 1 previous similar message [ 1289.241312] Lustre: Skipped 1 previous similar message [ 1291.850430] Lustre: server umount lustre-OST0000 complete [ 1299.431614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.438155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1299.450201] Lustre: Skipped 1 previous similar message [ 1299.467817] Lustre: Skipped 1 previous similar message [ 1302.604287] Lustre: server umount lustre-MDT0000 complete [ 1304.559632] LustreError: 38225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.588944] LustreError: 38225:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1306.918579] LustreError: 35037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388585 with bad export cookie 4425033911895354215 [ 1306.930455] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1307.376379] Lustre: server umount lustre-MDT0001 complete [ 1320.024206] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1323.737737] Key type lgssc unregistered [ 1324.037923] LNet: 42929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.046765] LNetError: 42929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.061429] LNet: Removed LNI 192.168.201.105@tcp [ 1324.963316] Key type .llcrypt unregistered [ 1324.965675] Key type ._llcrypt unregistered [ 1344.762804] Key type ._llcrypt registered [ 1344.765372] Key type .llcrypt registered [ 1344.894173] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 1360.118798] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1361.342763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1361.426101] alg: No test for adler32 (adler32-zlib) [ 1362.503701] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 1362.851592] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1364.615269] Key type lgssc registered [ 1365.905702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1405.695518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1405.750693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.173905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1407.247354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1407.370825] Lustre: lustre-MDT0000: new disk, initializing [ 1407.535529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1407.560217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1413.852985] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1424.923385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.982514] Lustre: 46086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1425.038755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1425.042649] Lustre: Skipped 1 previous similar message [ 1425.115744] Lustre: lustre-MDT0001: new disk, initializing [ 1425.176819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1425.226290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1425.231630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1430.256073] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1439.851242] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1446.104245] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1453.293023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1453.677627] Lustre: lustre-OST0000: new disk, initializing [ 1453.690491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1453.699644] Lustre: 48054:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1453.806916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1459.897911] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1460.271457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1460.288650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1460.410423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1468.877552] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1473.538613] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1473.683616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1478.679580] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1478.840564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1481.202320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1481.210211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.220754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1486.306174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1486.310497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1486.318306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1486.335571] Lustre: Skipped 1 previous similar message [ 1487.578869] Lustre: server umount lustre-OST0000 complete [ 1491.937202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1491.949817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1491.977923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.549140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.561159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.031292] Lustre: server umount lustre-MDT0000 complete [ 1502.215965] LustreError: 46076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388781 with bad export cookie 640883424526211859 [ 1502.227392] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1502.228734] LustreError: 46076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1502.556431] Lustre: server umount lustre-MDT0001 complete [ 1513.642910] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 04:53:11 (1787388791) [ 1520.413721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.954457] LustreError: 50513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1520.978957] LustreError: 50513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1521.069740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1525.906240] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1526.264745] LustreError: 50514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.293403] LustreError: 50514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1530.405895] LustreError: 50513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.145487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1534.536837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1538.988625] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1545.922908] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1551.673576] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1559.705056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1560.087396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1567.408299] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1574.323764] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1582.223844] LustreError: 53043:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1583.031341] Lustre: Modifying parameter general.timeout=1 in log params [ 1584.515063] Lustre: Modifying parameter general.timeout=3 in log params [ 1584.518029] Lustre: Skipped 1 previous similar message [ 1586.891599] Lustre: Modifying parameter general.timeout=6 in log params [ 1586.894782] Lustre: Skipped 2 previous similar messages [ 1591.547864] Lustre: Modifying parameter general.timeout=12 in log params [ 1591.554368] Lustre: Skipped 5 previous similar messages [ 1600.659206] Lustre: Modifying parameter general.timeout=22 in log params [ 1600.664209] Lustre: Skipped 9 previous similar messages [ 1617.657388] Lustre: Modifying parameter general.timeout=40 in log params [ 1617.666132] Lustre: Skipped 17 previous similar messages [ 1650.242752] Lustre: Modifying parameter general.timeout=74 in log params [ 1650.246108] Lustre: Skipped 33 previous similar messages [ 1712.225681] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 04:56:31 (1787388991) [ 1715.031537] Lustre: 59989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1715.801771] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1715.807608] Lustre: Skipped 58 previous similar messages [ 1716.676410] Lustre: 60087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1720.157927] Lustre: 60284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1720.170388] Lustre: 60284:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1750.871416] Lustre: 61861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1750.887501] Lustre: 61861:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1750.942523] LustreError: 61861:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1755.824092] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 04:57:14 (1787389034) [ 1769.955566] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1769.960522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1769.982046] Lustre: Skipped 2 previous similar messages [ 1770.011018] Lustre: Skipped 3 previous similar messages [ 1774.421388] Lustre: server umount lustre-MDT0000 complete [ 1775.074649] LustreError: 50514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1775.105100] LustreError: 50514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1778.499060] LustreError: 50495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389057 with bad export cookie 640883424526212741 [ 1778.513145] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1778.522635] LustreError: 50495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1778.818899] Lustre: server umount lustre-MDT0001 complete [ 1793.809362] Lustre: server umount lustre-OST0000 complete [ 1820.916466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.111327] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1821.305199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.673700] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1833.953838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.008173] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1834.023380] Lustre: 64394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1839.748472] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1848.926429] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1856.443219] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1864.778710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1865.050457] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1865.528637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.547660] Lustre: Skipped 1 previous similar message [ 1873.177813] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1881.216622] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1882.766141] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1882.782403] Lustre: Skipped 20 previous similar messages [ 1882.832084] Lustre: 66958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1892.838278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1892.844882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.861248] Lustre: Skipped 1 previous similar message [ 1892.864511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1895.466429] Lustre: server umount lustre-OST0000 complete [ 1901.024821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1901.030784] LustreError: Skipped 1 previous similar message [ 1905.653439] Lustre: server umount lustre-MDT0000 complete [ 1908.197257] LustreError: 64404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.222715] LustreError: 64404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1910.078137] LustreError: 65134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389189 with bad export cookie 640883424526215793 [ 1910.084422] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1910.086760] LustreError: 65134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1910.366739] Lustre: server umount lustre-MDT0001 complete [ 1922.461649] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1925.475670] Key type lgssc unregistered [ 1925.848320] LNet: 68165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.854089] LNetError: 68165:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1925.880892] LNet: Removed LNI 192.168.201.105@tcp [ 1927.010474] Key type .llcrypt unregistered [ 1927.012217] Key type ._llcrypt unregistered [ 1950.648831] Key type ._llcrypt registered [ 1950.652590] Key type .llcrypt registered [ 1950.776220] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1951.889404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1951.963631] alg: No test for adler32 (adler32-zlib) [ 1953.156697] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 1953.448809] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1955.193153] Key type lgssc registered [ 1956.405162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1963.869905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1963.890637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.296111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.648607] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1975.777360] LustreError: 69591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1975.788949] LustreError: 69591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1979.019073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.672814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1984.920209] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1993.647684] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1999.664255] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2007.218842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2007.993781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2015.471556] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2023.557815] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2032.675451] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2044.898443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.899271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2044.913989] Lustre: Skipped 2 previous similar messages [ 2044.927576] Lustre: Skipped 2 previous similar messages [ 2048.709102] Lustre: server umount lustre-MDT0000 complete [ 2050.020641] LustreError: 69591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.035226] LustreError: 69591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2051.887082] LustreError: 69572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389330 with bad export cookie 14478925376393432219 [ 2051.896917] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2051.903078] LustreError: 69572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2052.215194] Lustre: server umount lustre-MDT0001 complete [ 2066.099921] Lustre: server umount lustre-OST0000 complete [ 2090.554585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.789969] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2090.828118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2091.079122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.214858] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2105.316052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.458325] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2105.521766] Lustre: 74472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2106.091018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2111.126685] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2118.443653] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2123.865852] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2130.986102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2131.170692] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2131.610596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.288667] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2145.479679] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2147.551510] Lustre: 77047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2149.349405] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2154.977783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2155.004802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2155.006226] Lustre: Skipped 1 previous similar message [ 2155.006887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2155.023088] Lustre: Skipped 1 previous similar message [ 2160.101897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2160.105313] Lustre: Skipped 1 previous similar message [ 2160.795879] Lustre: server umount lustre-OST0000 complete [ 2165.218841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.231491] Lustre: Skipped 1 previous similar message [ 2165.236160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2165.243254] Lustre: Skipped 1 previous similar message [ 2170.336783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2170.350186] Lustre: Skipped 1 previous similar message [ 2171.558847] Lustre: server umount lustre-MDT0000 complete [ 2175.457475] LustreError: 75579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.494732] LustreError: 75579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2175.805735] LustreError: 74466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389454 with bad export cookie 14478925376393433066 [ 2175.813712] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2175.824564] LustreError: 74466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2176.576737] Lustre: server umount lustre-MDT0001 complete [ 2188.931757] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2192.061967] Key type lgssc unregistered [ 2192.373814] LNet: 78258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2192.382147] LNetError: 78258:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2192.398976] LNet: Removed LNI 192.168.201.105@tcp [ 2193.297409] Key type .llcrypt unregistered [ 2193.298515] Key type ._llcrypt unregistered [ 2216.848945] Key type ._llcrypt registered [ 2216.851299] Key type .llcrypt registered [ 2216.946497] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2218.130674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2218.222886] alg: No test for adler32 (adler32-zlib) [ 2219.291291] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 2219.582686] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2221.303182] Key type lgssc registered [ 2223.123796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2230.652782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2230.667671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.121742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.497493] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2242.534586] LustreError: 79671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.564938] LustreError: 79671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2244.329371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.774276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2249.433400] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2255.313223] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2260.596661] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2267.386457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2267.908405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.021947] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2284.149400] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2289.825565] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2290.825298] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2292.870333] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2292.881157] Lustre: Skipped 1 previous similar message [ 2294.994220] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2295.006486] Lustre: Skipped 1 previous similar message [ 2300.049122] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2300.060463] Lustre: Skipped 4 previous similar messages [ 2308.510987] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2308.526435] Lustre: Skipped 7 previous similar messages [ 2325.106076] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2325.109617] Lustre: Skipped 13 previous similar messages [ 2357.397551] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2357.403873] Lustre: Skipped 33 previous similar messages [ 2394.495303] Lustre: 87777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2396.875968] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:07:55 (1787389675) [ 2398.556617] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2400.144789] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:07:59 (1787389679) [ 2405.346574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2405.351782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.362697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2405.384587] Lustre: Skipped 1 previous similar message [ 2405.396352] Lustre: Skipped 1 previous similar message [ 2410.468115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2410.482665] Lustre: Skipped 1 previous similar message [ 2415.589027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.603629] Lustre: Skipped 1 previous similar message [ 2418.145098] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2418.358254] Lustre: server umount lustre-OST0000 complete [ 2423.778833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2423.789410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.814913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.826246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.702921] Lustre: server umount lustre-MDT0000 complete [ 2430.953811] LustreError: 81629:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.987117] LustreError: 81629:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2432.468600] LustreError: 79650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389711 with bad export cookie 1941812497038458821 [ 2432.479773] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2432.480246] LustreError: 79650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2432.823543] Lustre: server umount lustre-MDT0001 complete [ 2444.333674] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2446.964138] Key type lgssc unregistered [ 2447.361789] LNet: 89026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.413160] LNetError: 89026:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2447.429216] LNet: Removed LNI 192.168.201.105@tcp [ 2448.208984] Key type .llcrypt unregistered [ 2448.210732] Key type ._llcrypt unregistered [ 2452.792641] Key type ._llcrypt registered [ 2452.795621] Key type .llcrypt registered [ 2452.946455] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2457.508502] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules [ 2458.773783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2458.946958] alg: No test for adler32 (adler32-zlib) [ 2459.962979] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 2460.227411] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2461.913167] Key type lgssc registered [ 2463.573191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2465.557296] LustreError: 90270:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2486.110170] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2488.960429] LustreError: 90270:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2496.743755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2496.774436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.443891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.621779] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2508.777055] LustreError: 91302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.805971] LustreError: 91302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2512.639907] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:09:51 (1787389791) [ 2513.902990] LustreError: 91301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.535960] LustreError: 91333:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2517.554268] LustreError: 91333:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2517.560727] LustreError: 91333:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2517.981815] Lustre: server umount lustre-MDT0000 complete [ 2534.584868] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2538.094403] Key type lgssc unregistered [ 2538.427889] LNet: 92836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2538.437032] LNetError: 92836:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2538.464297] LNet: Removed LNI 192.168.201.105@tcp [ 2539.862286] Key type .llcrypt unregistered [ 2539.867334] Key type ._llcrypt unregistered [ 2562.433551] Key type ._llcrypt registered [ 2562.436039] Key type .llcrypt registered [ 2562.573930] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2564.159398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2564.311819] alg: No test for adler32 (adler32-zlib) [ 2565.669634] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 2566.108209] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2567.815386] Key type lgssc registered [ 2569.248216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2577.692690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2577.726887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.232569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.156553] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2589.671733] LustreError: 94058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.693830] LustreError: 94058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2593.338433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2594.425693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2599.736409] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2608.675204] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2615.297497] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2623.271586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2624.029631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2631.143354] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2638.994706] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2686.950841] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:12:45 (1787389965) [ 2688.633672] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2690.661277] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:12:49 (1787389969) [ 2695.653777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2695.667199] Lustre: Skipped 1 previous similar message [ 2695.672483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2695.679825] Lustre: Skipped 2 previous similar messages [ 2700.706106] Lustre: server umount lustre-MDT0000 complete [ 2700.777195] LustreError: 96189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.797228] LustreError: 96189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2704.889092] LustreError: 94037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389983 with bad export cookie 3867865118260904169 [ 2704.896453] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.901241] LustreError: 94037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2705.232390] Lustre: server umount lustre-MDT0001 complete [ 2718.709847] Lustre: server umount lustre-OST0000 complete [ 2732.916575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2733.231121] LustreError: 98332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.388454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.671722] LustreError: 98333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.825204] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2743.780094] LustreError: 98332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.156887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.033587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2753.598664] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2761.619162] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2768.007296] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2782.818727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2783.055973] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2783.061743] LustreError: 98320:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2783.080047] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2783.089293] LustreError: 100417:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2783.103988] LustreError: 100417:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2783.110817] LustreError: 100417:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2783.219667] Lustre: server umount lustre-OST0000 complete [ 2783.228904] LustreError: 100417:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2793.149786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2793.366141] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2793.374164] LustreError: 98321:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2793.396379] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2793.404367] LustreError: 100882:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2793.416578] LustreError: 100882:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2793.423347] LustreError: 100882:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2793.518199] Lustre: server umount lustre-OST0000 complete [ 2793.523302] LustreError: 100882:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2804.173759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2804.357902] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2804.367503] Lustre: Found index 0 for lustre-OST0000, updating log [ 2804.377866] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2804.500869] Lustre: lustre-OST0000: new disk, initializing [ 2804.506425] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2804.513245] Lustre: 101417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2804.827798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2812.303394] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2817.086245] Lustre: Failing over lustre-OST0000 [ 2817.325310] Lustre: server umount lustre-OST0000 complete [ 2823.137215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.141349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2823.146491] Lustre: Skipped 1 previous similar message [ 2823.165923] Lustre: Skipped 1 previous similar message [ 2827.917259] Lustre: server umount lustre-MDT0000 complete [ 2828.264341] LustreError: 99063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.280525] LustreError: 99063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2831.935545] LustreError: 98314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390110 with bad export cookie 3867865118260906913 [ 2831.945523] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2832.498554] Lustre: server umount lustre-MDT0001 complete [ 2842.304686] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:15:20 (1787390120) [ 2858.989740] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2871.476726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2872.132115] LustreError: 103762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2872.259485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2876.417315] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2885.468070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2891.511985] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2895.135938] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2903.020809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2910.511655] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2919.821520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2919.924793] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2920.003751] Lustre: lustre-OST0001: new disk, initializing [ 2920.006907] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2920.011289] Lustre: 106078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2920.259252] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2920.262842] Lustre: Skipped 2 previous similar messages [ 2926.720196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2926.727929] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2926.857345] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2927.621087] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2936.216621] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2945.273169] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2950.917528] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 2952.677150] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2952.682448] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2955.817135] Lustre: server umount lustre-MDT0001 complete [ 2956.041028] LustreError: 103759:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.086956] LustreError: 103759:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2967.257049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2967.394634] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2967.396561] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2967.437249] Lustre: 103751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2968.557464] LustreError: 107615:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2968.566553] LustreError: 107615:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2968.579967] LustreError: 103754:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2968.738327] LustreError: 107621:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.105@tcp: cfg command failed: rc = -17 [ 2968.749766] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2968.749766] [ 2973.198304] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2974.014451] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2984.709433] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:17:43 (1787390263) [ 3002.676041] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3020.407256] Lustre: 109691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3048.135177] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3051.505806] Lustre: 110827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3141.608756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3141.625472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3141.640588] Lustre: Skipped 2 previous similar messages [ 3141.648301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3141.660034] Lustre: Skipped 3 previous similar messages [ 3146.591371] Lustre: server umount lustre-MDT0000 complete [ 3147.239520] LustreError: 110841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.254413] LustreError: 110841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3150.901185] LustreError: 103743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390429 with bad export cookie 3867865118260907907 [ 3150.904075] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3150.913719] LustreError: 103743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3151.344447] Lustre: server umount lustre-MDT0001 complete [ 3165.973569] Lustre: server umount lustre-OST0000 complete [ 3168.480096] Lustre: 93369:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390431/real 1787390431] req@ffff8f0d42a29500 x1874214094471680/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390447 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3168.527325] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3168.534183] Lustre: Skipped 3 previous similar messages [ 3170.210908] Lustre: server umount lustre-OST0001 complete [ 3176.159947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3186.814464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3196.453095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3207.381195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3228.120494] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3240.528674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3240.548526] Lustre: lustre-MDT0000: reset Object Index mappings [ 3241.170027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3241.173299] Lustre: Skipped 1 previous similar message [ 3246.290647] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3256.038878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.069133] Lustre: lustre-MDT0001: reset Object Index mappings [ 3261.800901] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3265.372412] Lustre: 115930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3272.369029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3279.943232] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3281.529331] LustreError: 116286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.552249] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3281.556279] LustreError: 116286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3286.728606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3289.508717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3295.237484] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3295.241903] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3297.782769] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3307.095458] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3311.226885] Lustre: 117809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3362.936492] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:24:01 (1787390641) [ 3389.920296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3389.932524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.949964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3389.957358] Lustre: Skipped 4 previous similar messages [ 3395.120521] Lustre: server umount lustre-MDT0000 complete [ 3399.183106] LustreError: 114764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390678 with bad export cookie 3867865118260954758 [ 3399.210044] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3402.729575] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3402.746735] Lustre: Skipped 4 previous similar messages [ 3405.933707] Lustre: server umount lustre-MDT0001 complete [ 3415.910817] Lustre: server umount lustre-OST0000 complete [ 3426.777794] Lustre: server umount lustre-OST0001 complete [ 3433.738987] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3446.077312] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3483.247305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.534834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3483.566045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3483.678512] Lustre: lustre-MDT0000: new disk, initializing [ 3483.780033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3483.787259] Lustre: Skipped 3 previous similar messages [ 3483.814164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3488.159843] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3494.981687] Lustre: server umount lustre-MDT0000 complete [ 3505.341679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.901119] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3505.919090] Lustre: 122077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3505.953757] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3512.439316] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3518.355477] Lustre: server umount lustre-MDT0000 complete [ 3528.326843] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:26:46 (1787390806) [ 3530.413848] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3532.563216] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:26:51 (1787390811) [ 3552.114145] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3563.971535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3569.929887] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3580.261564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3580.468607] Lustre: 124127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3580.535499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3580.540700] Lustre: Skipped 1 previous similar message [ 3580.671436] Lustre: lustre-MDT0001: new disk, initializing [ 3580.843185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3580.874182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3586.692541] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3591.896310] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3599.460297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3599.768187] Lustre: lustre-OST0000: new disk, initializing [ 3599.770797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3599.777870] Lustre: 125706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3601.193718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3601.213238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3601.366980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3607.751176] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3619.600551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3619.836562] Lustre: lustre-OST0001: new disk, initializing [ 3619.846598] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3619.876909] Lustre: 126579:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3621.573913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3621.595877] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3621.774085] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3628.056504] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3637.745540] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3641.390886] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4042.223470] Lustre: 132996:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4042.241576] Lustre: 132996:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4042.254454] Lustre: 132996:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4042.263970] Lustre: 132996:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4042.272818] Lustre: 132996:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4042.276974] Lustre: 132996:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4327.510697] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:40:04 (1787391604) [ 4343.779204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4343.785865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4343.795569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4343.795582] Lustre: Skipped 2 previous similar messages [ 4343.832527] Lustre: Skipped 7 previous similar messages [ 4344.554454] Lustre: server umount lustre-MDT0000 complete [ 4348.821936] LustreError: 124122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391627 with bad export cookie 3867865118261022560 [ 4348.831192] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4348.844907] LustreError: 124122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4348.911917] LustreError: 127388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4348.925843] LustreError: 127388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 4348.927443] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.962301] Lustre: Skipped 2 previous similar messages [ 4348.977237] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4348.988867] Lustre: Skipped 4 previous similar messages [ 4354.023169] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4354.044219] Lustre: Skipped 1 previous similar message [ 4363.231267] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4363.235855] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4363.255925] Lustre: Skipped 3 previous similar messages [ 4363.561836] Lustre: server umount lustre-MDT0001 complete [ 4382.175168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4382.526157] Lustre: server umount lustre-OST0000 complete [ 4393.587324] Lustre: server umount lustre-OST0001 complete [ 4403.941808] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 4416.379241] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4463.955462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.366588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4464.415109] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4464.521353] Lustre: lustre-MDT0000: new disk, initializing [ 4464.587579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4464.592013] Lustre: Skipped 5 previous similar messages [ 4464.613459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4469.377894] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4484.987565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4485.368935] Lustre: 136051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4485.485701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4485.492060] Lustre: Skipped 1 previous similar message [ 4485.624666] Lustre: lustre-MDT0001: new disk, initializing [ 4485.830059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4485.858855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4492.678979] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4503.617734] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4510.664328] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4520.171273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4520.506956] Lustre: lustre-OST0000: new disk, initializing [ 4520.511417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4520.527424] Lustre: 138018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4520.653798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4520.656935] Lustre: Skipped 1 previous similar message [ 4522.498721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4522.515915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4522.698744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4528.436785] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4539.577406] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4550.863403] Lustre: lustre-MDD0000: changelog on [ 4554.209787] Lustre: lustre-MDD0001: changelog on [ 6322.422902] Lustre: lustre-MDD0001: changelog off [ 6323.873244] Lustre: 138025:0:(llog_cat.c:984:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6324.638345] Lustre: lustre-MDD0000: changelog off [ 6329.835895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6329.843461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.855316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6330.339971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6330.352265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6335.274603] Lustre: server umount lustre-OST0000 complete [ 6339.040902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6339.046905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6339.065246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6339.069556] Lustre: Skipped 2 previous similar messages [ 6344.808839] Lustre: server umount lustre-MDT0000 complete [ 6345.698338] LustreError: 136063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6345.731148] LustreError: 136063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6348.338704] LustreError: 136043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393627 with bad export cookie 3867865118261797719 [ 6348.360753] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6348.912227] Lustre: server umount lustre-MDT0001 complete [ 6360.940239] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6364.210293] Key type lgssc unregistered [ 6364.625893] LNet: 147145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6364.635291] LNetError: 147145:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6364.657361] LNet: Removed LNI 192.168.201.105@tcp [ 6365.527519] Key type .llcrypt unregistered [ 6365.530159] Key type ._llcrypt unregistered [ 6386.298039] Key type ._llcrypt registered [ 6386.302136] Key type .llcrypt registered [ 6386.412835] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6401.180085] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6402.737567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6402.778803] alg: No test for adler32 (adler32-zlib) [ 6403.794367] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 6404.075167] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6405.703254] Key type lgssc registered [ 6406.879773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6440.283594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6440.345534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.530475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6441.550321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6441.605631] Lustre: lustre-MDT0000: new disk, initializing [ 6441.650734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6441.658471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6446.121077] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6455.716355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6455.849950] Lustre: 150345:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6455.919211] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6455.925932] Lustre: Skipped 1 previous similar message [ 6456.070444] Lustre: lustre-MDT0001: new disk, initializing [ 6456.207585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6456.238733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6456.251790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6460.552471] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6469.383253] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6475.484606] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6483.656575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6484.064126] Lustre: lustre-OST0000: new disk, initializing [ 6484.070952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6484.080252] Lustre: 152314:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6484.171832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6491.597333] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6494.299112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6494.306911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6494.457897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6500.598396] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6506.486730] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_module obdecho/obdecho [ 6508.062248] LustreError: 153317:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6508.071225] LustreError: 153317:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6509.577900] LustreError: 153413:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6520.289916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6520.304730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6520.314294] Lustre: Skipped 1 previous similar message [ 6525.421664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6525.424268] Lustre: Skipped 3 previous similar messages [ 6526.179501] Lustre: server umount lustre-MDT0000 complete [ 6529.827645] LustreError: 150336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393808 with bad export cookie 10117668462271961422 [ 6529.840694] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6529.848762] LustreError: 150336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6530.062274] Lustre: server umount lustre-MDT0001 complete [ 6543.383067] Lustre: server umount lustre-OST0000 complete [ 6562.331981] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6572.994806] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6607.315562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6607.556468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6607.590100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6607.666316] Lustre: lustre-MDT0000: new disk, initializing [ 6607.741612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6607.755799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6611.709806] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6621.524331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6621.668548] Lustre: 157131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6621.705750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6621.708843] Lustre: Skipped 1 previous similar message [ 6621.779671] Lustre: lustre-MDT0001: new disk, initializing [ 6621.916485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6621.977138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6621.987480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6626.250539] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6634.918934] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6639.831767] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6645.987232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6646.276906] Lustre: lustre-OST0000: new disk, initializing [ 6646.279699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6646.284472] Lustre: 159097:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6646.351084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6652.985627] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6655.577918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6655.592775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6655.693314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6662.706333] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6667.864286] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6668.175256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6673.280933] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6673.497724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6676.449173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6676.461892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.473378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6676.476462] Lustre: Skipped 2 previous similar messages [ 6676.490182] Lustre: Skipped 1 previous similar message [ 6681.585141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6681.599784] Lustre: Skipped 1 previous similar message [ 6681.781565] Lustre: server umount lustre-OST0000 complete [ 6686.690078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6686.692984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6686.701656] Lustre: Skipped 1 previous similar message [ 6691.669971] Lustre: server umount lustre-MDT0000 complete [ 6695.162401] LustreError: 157122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393974 with bad export cookie 10117668462271962381 [ 6695.181337] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6695.672153] Lustre: server umount lustre-MDT0001 complete [ 6699.370717] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:19:38 (1787393978) [ 6700.768738] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6702.437339] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:19:41 (1787393981) [ 6710.053924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6710.694278] LustreError: 161510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6710.720377] LustreError: 161510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6710.841041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6715.064818] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6715.875948] LustreError: 161511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6721.002759] LustreError: 161510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6723.107891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6728.067651] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6736.249839] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6742.589548] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6750.004537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6750.287798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6750.294122] Lustre: Skipped 1 previous similar message [ 6756.659961] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6763.541124] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6774.531176] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 6774.534395] Lustre: Skipped 1 previous similar message [ 6774.752436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6774.770204] LustreError: Skipped 1 previous similar message [ 6774.776682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.162442] Lustre: server umount lustre-MDT0000 complete [ 6778.736086] LustreError: 161491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394057 with bad export cookie 10117668462271963263 [ 6778.747918] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6778.756740] LustreError: 161491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6778.849907] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6778.865808] LustreError: 161507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6778.871625] Lustre: Skipped 1 previous similar message [ 6783.971279] LustreError: 161506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6783.989488] LustreError: 161506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6793.183492] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6793.187084] LustreError: 163627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6793.187210] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6793.187216] Lustre: Skipped 7 previous similar messages [ 6793.264433] LustreError: 163627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6793.712851] Lustre: server umount lustre-MDT0001 complete [ 6800.529042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6812.619810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6813.008875] LustreError: 165352:0:(llog.c:1655:llog_backup()) MGC192.168.201.105@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6813.017499] Lustre: 165352:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.105@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6820.587144] LustreError: 165357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6820.600109] LustreError: 165357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6853.605830] LustreError: 165374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6853.635147] LustreError: 165374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 6873.834198] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6873.841439] Lustre: 165352:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.105@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6914.558854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6919.320492] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6919.657187] LustreError: 165357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6919.667631] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6919.672350] LustreError: 165357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 43 previous similar messages [ 6927.672865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6944.226778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6953.008121] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6953.974576] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6959.448965] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6961.356650] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6967.119333] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6968.497443] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6984.677557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6984.685499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6984.688092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6984.699050] Lustre: Skipped 2 previous similar messages [ 6985.654417] Lustre: server umount lustre-MDT0000 complete [ 6989.944039] LustreError: 161491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394268 with bad export cookie 10117668462271969752 [ 6989.948894] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6989.961274] LustreError: 161491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6990.403842] Lustre: server umount lustre-MDT0001 complete [ 7005.582727] Lustre: server umount lustre-OST0000 complete [ 7028.005659] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7039.936629] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7082.970698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7083.237670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7083.280168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7083.384478] Lustre: lustre-MDT0000: new disk, initializing [ 7083.454504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7083.457307] Lustre: Skipped 1 previous similar message [ 7083.473776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7088.900708] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7101.290055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7101.410496] Lustre: 170461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7101.735158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7105.789786] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7115.396436] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7121.515491] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7128.895503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7129.221745] Lustre: lustre-OST0000: new disk, initializing [ 7129.224980] Lustre: Skipped 1 previous similar message [ 7129.229721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7129.235116] Lustre: Skipped 2 previous similar messages [ 7129.240064] Lustre: 172430:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7135.285909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7135.297797] Lustre: Skipped 1 previous similar message [ 7135.306338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7135.516139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7137.460649] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7149.610401] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7155.828947] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7156.095912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7161.669174] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7161.943965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7166.440964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7166.444398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7166.463025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7166.463039] Lustre: Skipped 2 previous similar messages [ 7166.496271] Lustre: Skipped 1 previous similar message [ 7170.541852] Lustre: server umount lustre-OST0000 complete [ 7181.360304] Lustre: server umount lustre-MDT0000 complete [ 7181.792992] LustreError: 170474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7181.816432] LustreError: 170474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 7185.451465] LustreError: 170455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394464 with bad export cookie 10117668462271986559 [ 7185.469715] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7185.476017] LustreError: 170455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7185.901391] Lustre: server umount lustre-MDT0001 complete [ 7194.863650] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:27:53 (1787394473) [ 7201.447646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.097375] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7215.563969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7220.584776] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7229.026849] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7235.235529] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7242.204044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.547510] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7262.231591] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7272.515258] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:29:11 (1787394551) [ 7278.562107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7278.568633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7278.576593] Lustre: Skipped 2 previous similar messages [ 7283.827600] Lustre: server umount lustre-OST0000 complete [ 7297.001545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7297.008367] Lustre: Skipped 10 previous similar messages [ 7302.117848] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7302.372233] Lustre: server umount lustre-MDT0000 complete [ 7306.751628] LustreError: 174823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394585 with bad export cookie 10117668462271987441 [ 7306.764163] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7307.100206] Lustre: server umount lustre-MDT0001 complete [ 7318.668279] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7322.387904] Key type lgssc unregistered [ 7322.859744] LNet: 178572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7322.874669] LNetError: 178572:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7322.910117] LNet: Removed LNI 192.168.201.105@tcp [ 7324.060262] Key type .llcrypt unregistered [ 7324.063730] Key type ._llcrypt unregistered [ 7345.406958] Key type ._llcrypt registered [ 7345.409153] Key type .llcrypt registered [ 7345.528572] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7347.121988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7347.252281] alg: No test for adler32 (adler32-zlib) [ 7348.511776] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 7348.987909] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7350.756708] Key type lgssc registered [ 7352.431652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7361.949274] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7361.996055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7377.761374] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7382.879870] LustreError: 179818:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 7387.999634] LustreError: 179818:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7388.018320] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7388.029559] LustreError: MGC192.168.201.105@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7388.046118] LustreError: 179818:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7388.061349] LustreError: 179818:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7388.070861] LustreError: 179818:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7388.155532] Lustre: server umount lustre-OST0000 complete [ 7388.157633] LustreError: 179818:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7398.523995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7399.897232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7404.214987] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7409.137878] LustreError: 180317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7409.173714] LustreError: 180317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7413.309303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7413.785281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7419.286656] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7426.351671] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7431.169462] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7438.247942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7438.321254] LustreError: 182136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7438.432650] LustreError: 182132:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7438.447071] Lustre: 182132:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.105@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7438.573890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7444.931615] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7453.284244] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7456.245245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7456.248632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7456.252521] Lustre: Skipped 2 previous similar messages [ 7456.264014] Lustre: Skipped 2 previous similar messages [ 7461.357565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7461.377113] Lustre: Skipped 2 previous similar messages [ 7461.915394] Lustre: server umount lustre-MDT0000 complete [ 7465.316877] LustreError: 180297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394744 with bad export cookie 16406277998681323388 [ 7465.328270] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7465.335248] LustreError: 180297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7465.710427] Lustre: server umount lustre-MDT0001 complete [ 7480.165587] Lustre: server umount lustre-OST0000 complete [ 7490.597643] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7501.086926] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7534.959767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7535.337383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7535.363843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7535.439918] Lustre: lustre-MDT0000: new disk, initializing [ 7535.513689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7535.527748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7540.367990] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7550.878491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7550.965036] Lustre: 185661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7551.050072] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7551.053989] Lustre: Skipped 1 previous similar message [ 7551.193023] Lustre: lustre-MDT0001: new disk, initializing [ 7551.305439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7551.341032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7551.359280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7556.035297] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7566.521514] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7572.844988] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7580.647320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7581.082860] Lustre: lustre-OST0000: new disk, initializing [ 7581.088169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7581.094192] Lustre: 187625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7581.226319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7588.407933] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7588.924475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7588.934768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7589.139074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7598.492485] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7602.646889] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7602.864687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7607.617091] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7607.824997] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7609.825612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7609.828681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7609.838654] LustreError: Skipped 1 previous similar message [ 7609.843835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7609.853741] Lustre: Skipped 1 previous similar message [ 7609.875575] Lustre: Skipped 1 previous similar message [ 7614.961966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7614.966510] Lustre: Skipped 1 previous similar message [ 7620.081069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7620.087392] Lustre: Skipped 1 previous similar message [ 7624.160357] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7624.295882] Lustre: server umount lustre-OST0000 complete [ 7630.305104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7630.318287] Lustre: Skipped 1 previous similar message [ 7630.324562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7630.334095] Lustre: Skipped 1 previous similar message [ 7636.174483] Lustre: server umount lustre-MDT0000 complete [ 7640.554428] LustreError: 185672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7640.582797] LustreError: 185672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7641.168091] LustreError: 185653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394920 with bad export cookie 16406277998681324109 [ 7641.168873] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7641.181676] LustreError: 185653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7641.653637] Lustre: server umount lustre-MDT0001 complete [ 7652.116897] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:35:30 (1787394930) [ 7668.990503] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7671.950954] Key type lgssc unregistered [ 7672.270971] LNet: 190575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7672.279779] LNetError: 190575:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7673.318452] LNet: Removed LNI 192.168.201.105@tcp [ 7674.415176] Key type .llcrypt unregistered [ 7674.417641] Key type ._llcrypt unregistered [ 7693.553720] Key type ._llcrypt registered [ 7693.555253] Key type .llcrypt registered [ 7693.648961] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7695.052950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7695.106429] alg: No test for adler32 (adler32-zlib) [ 7696.393468] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 7696.631290] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7698.336694] Key type lgssc registered [ 7699.743736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7708.113051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7708.128280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7709.627245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7714.497075] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7719.904281] LustreError: 191792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7719.914053] LustreError: 191792:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7723.512523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7723.909910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7728.706670] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7736.961665] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7743.303386] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7751.201040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7751.432880] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7751.440398] LustreError: 193659:0:(llog.c:1696:llog_backup()) MGC192.168.201.105@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7751.452487] Lustre: 193659:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.105@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7751.600205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7758.121982] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7767.173835] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7770.080348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7770.088701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7770.117240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7774.179223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7774.189718] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7776.119027] Lustre: server umount lustre-MDT0000 complete [ 7779.297616] LustreError: 191792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7779.317308] LustreError: 191792:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7781.465706] LustreError: 191772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395060 with bad export cookie 13644309306459431589 [ 7781.482745] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7781.492516] LustreError: 191772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7781.934808] Lustre: server umount lustre-MDT0001 complete [ 7796.710415] Lustre: server umount lustre-OST0000 complete [ 7807.336495] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7819.634880] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7858.925427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7859.237295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7859.279428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7859.413158] Lustre: lustre-MDT0000: new disk, initializing [ 7859.500578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7859.517387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7864.015316] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7876.305269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7876.573207] Lustre: 197181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7876.676267] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7876.677948] Lustre: Skipped 1 previous similar message [ 7876.679654] LustreError: 197964:0:(llog.c:1696:llog_backup()) MGC192.168.201.105@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7876.689861] LustreError: 197964:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7876.693425] Lustre: 197964:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.105@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7876.731934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7876.741663] Lustre: Skipped 1 previous similar message [ 7876.846080] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7876.859967] Lustre: Skipped 2 previous similar messages [ 7876.903247] Lustre: lustre-MDT0001: new disk, initializing [ 7877.023600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7877.095371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7877.112904] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7883.431157] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7895.112259] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7902.188668] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7909.478657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7909.760246] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7909.762413] Lustre: Skipped 1 previous similar message [ 7909.765897] LustreError: 199126:0:(llog.c:1696:llog_backup()) MGC192.168.201.105@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7909.773305] LustreError: 199126:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7909.781116] Lustre: 199126:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.105@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7909.803851] Lustre: 199126:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7909.847787] Lustre: lustre-OST0000: new disk, initializing [ 7909.853868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7909.871115] Lustre: 199150:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7909.988371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7915.070748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7915.080818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7915.183127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7916.888656] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7927.051823] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7931.939393] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7932.179754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7937.303269] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7937.571830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7941.090217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7941.094138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7941.113516] Lustre: Skipped 1 previous similar message [ 7941.119399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7941.144048] Lustre: Skipped 1 previous similar message [ 7945.929810] Lustre: server umount lustre-OST0000 complete [ 7951.330076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7951.343221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7951.354365] Lustre: Skipped 2 previous similar messages [ 7951.377454] Lustre: Skipped 1 previous similar message [ 7961.569289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7961.588484] Lustre: Skipped 3 previous similar messages [ 7964.127310] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7964.509663] Lustre: server umount lustre-MDT0000 complete [ 7966.696176] LustreError: 197193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7968.464222] LustreError: 199149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395247 with bad export cookie 13644309306459432310 [ 7968.471292] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7968.485833] LustreError: 199149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7968.838283] Lustre: server umount lustre-MDT0001 complete [ 7977.846798] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:40:56 (1787395256) [ 7996.359375] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8007.860385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8008.391942] LustreError: 202277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8008.419792] LustreError: 202277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8008.472428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8013.377160] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8013.798693] LustreError: 202278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8021.788950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8026.825525] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8030.039821] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8037.619261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8045.752662] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8056.256853] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8056.524580] Lustre: lustre-OST0001: new disk, initializing [ 8056.528454] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8056.538566] Lustre: 204590:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8056.756205] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8056.768677] Lustre: Skipped 2 previous similar messages [ 8063.930852] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8064.663321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8064.670368] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8064.824684] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8075.223614] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8083.958921] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8086.238834] Lustre: DEBUG MARKER: ADD OST3 [ 8091.768187] loop: module loaded [ 8091.878108] loop0: detected capacity change from 0 to 204800000 [ 8092.180370] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8092.198959] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8092.207353] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8098.970601] loop0: detected capacity change from 0 to 204800000 [ 8099.622577] Lustre: DEBUG MARKER: STOP OST3 [ 8107.005442] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8107.335822] Lustre: lustre-OST0003: new disk, initializing [ 8107.354850] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8107.368615] Lustre: 206618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8108.771085] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8108.946135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8108.959875] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8109.002041] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8110.818594] LustreError: 206724:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8111.784013] Lustre: Failing over lustre-OST0003 [ 8112.031832] Lustre: server umount lustre-OST0003 complete [ 8113.938892] LustreError: 203767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8113.951038] LustreError: 203767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8114.154293] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8114.157353] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.162272] Lustre: Skipped 1 previous similar message [ 8115.389491] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8124.086711] Lustre: DEBUG MARKER: START OST3 again [ 8131.435461] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8131.887903] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8131.900151] Lustre: Skipped 1 previous similar message [ 8131.920554] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8133.669253] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8133.786611] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8133.786975] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8138.376518] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8158.213949] Lustre: Failing over lustre-OST0003 [ 8158.283706] Lustre: server umount lustre-OST0003 complete [ 8159.713093] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8159.719791] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8159.724621] LustreError: Skipped 1 previous similar message [ 8159.739248] LustreError: 203765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8159.739258] LustreError: 203765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 8178.152204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.181291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8178.183878] Lustre: Skipped 4 previous similar messages [ 8178.198622] Lustre: Skipped 3 previous similar messages [ 8182.535913] Lustre: server umount lustre-MDT0000 complete [ 8186.519972] LustreError: 204588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395465 with bad export cookie 13644309306459433192 [ 8186.529014] LustreError: 204588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8186.530467] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8193.034736] Lustre: server umount lustre-MDT0001 complete [ 8197.406953] Lustre: server umount lustre-OST0000 complete [ 8201.447976] Lustre: server umount lustre-OST0001 complete [ 8210.152925] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8219.188741] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8259.242235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8259.428708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8259.446665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8259.488417] Lustre: lustre-MDT0000: new disk, initializing [ 8259.555263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8265.685506] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8276.855201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8276.954887] Lustre: 212049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8277.017820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8277.020777] Lustre: Skipped 1 previous similar message [ 8277.087437] Lustre: lustre-MDT0001: new disk, initializing [ 8277.235216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8277.247842] Lustre: Skipped 1 previous similar message [ 8277.284769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8277.293874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8282.212952] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8291.374950] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8297.209644] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8304.041289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8304.414529] Lustre: 214018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8306.072961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8306.210317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8311.542391] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8321.219505] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8326.706197] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8326.970267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8331.443393] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8331.620194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8336.876840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8336.882555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8336.890181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8336.897958] Lustre: Skipped 3 previous similar messages [ 8336.910092] Lustre: Skipped 4 previous similar messages [ 8339.887484] Lustre: server umount lustre-OST0000 complete [ 8349.964934] Lustre: server umount lustre-MDT0000 complete [ 8352.225988] LustreError: 212919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8352.242830] LustreError: 212919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 8353.893089] LustreError: 212042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395632 with bad export cookie 13644309306459435488 [ 8353.897168] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8353.901257] LustreError: 212042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8354.316341] Lustre: server umount lustre-MDT0001 complete [ 8357.752966] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 06:47:16 (1787395636) [ 8372.967455] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8383.557769] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8424.113881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8424.420941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8424.450701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8424.455234] Lustre: Skipped 1 previous similar message [ 8424.593729] Lustre: lustre-MDT0000: new disk, initializing [ 8424.599557] Lustre: Skipped 1 previous similar message [ 8424.734283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8424.745610] Lustre: Skipped 1 previous similar message [ 8429.899222] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8439.570918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8439.667754] Lustre: 218630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8439.859602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8444.696863] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8453.105801] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8457.917482] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8464.098795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8464.323657] Lustre: 220599:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8465.897760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8469.848670] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8478.327381] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8482.589653] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8482.737037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8486.634900] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8486.820640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8491.492971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8491.493808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8491.505285] Lustre: Skipped 2 previous similar messages [ 8491.520715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8491.521201] LustreError: Skipped 3 previous similar messages [ 8491.535118] Lustre: Skipped 4 previous similar messages [ 8494.713633] Lustre: server umount lustre-OST0000 complete [ 8504.814558] Lustre: server umount lustre-MDT0000 complete [ 8506.853199] LustreError: 218640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8506.888981] LustreError: 218640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8508.396220] LustreError: 218621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395787 with bad export cookie 13644309306459436370 [ 8508.404408] LustreError: 218621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8508.406815] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8508.777322] Lustre: server umount lustre-MDT0001 complete [ 8515.478975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8519.741171] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8526.401438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8531.332865] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8537.473614] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8542.031256] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8547.089937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8547.347517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8547.351506] Lustre: Skipped 6 previous similar messages [ 8552.322937] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8558.996977] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8570.853562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8570.871478] LustreError: Skipped 1 previous similar message [ 8574.350700] Lustre: server umount lustre-OST0000 complete [ 8583.500907] Lustre: server umount lustre-MDT0000 complete [ 8585.941475] LustreError: 222795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395864 with bad export cookie 13644309306459437252 [ 8585.947245] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8585.949916] LustreError: 222795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8586.197334] Lustre: server umount lustre-MDT0001 complete [ 8696.275304] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8699.040783] Key type lgssc unregistered [ 8699.360827] LNet: 226481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8699.372141] LNetError: 226481:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8699.393991] LNet: Removed LNI 192.168.201.105@tcp [ 8700.411196] Key type .llcrypt unregistered [ 8700.412825] Key type ._llcrypt unregistered [ 8724.829318] Key type ._llcrypt registered [ 8724.831811] Key type .llcrypt registered [ 8724.928370] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8741.819395] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8743.478298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8743.521475] alg: No test for adler32 (adler32-zlib) [ 8744.606665] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 8744.802697] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8746.447214] Key type lgssc registered [ 8748.087835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8780.303587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8780.321273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8781.659335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8781.719694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8781.857989] Lustre: lustre-MDT0000: new disk, initializing [ 8781.973557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8782.001834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8786.300193] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8795.144794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8795.276170] Lustre: 229834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8795.319925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8795.322462] Lustre: Skipped 1 previous similar message [ 8795.376963] Lustre: lustre-MDT0001: new disk, initializing [ 8795.457510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8795.489117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8795.499203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8799.453440] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8808.343498] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8814.049885] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8820.235555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8820.580121] Lustre: lustre-OST0000: new disk, initializing [ 8820.590851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8820.596672] Lustre: 231800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8820.696070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8826.953259] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8828.513319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8828.532043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8828.709643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8835.888248] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8840.541127] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8840.715425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8845.361732] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8845.593915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8849.378171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8849.381979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8849.409668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8853.914738] Lustre: server umount lustre-OST0000 complete [ 8859.616548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8859.628703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8859.632738] Lustre: Skipped 2 previous similar messages [ 8859.644898] Lustre: Skipped 2 previous similar messages [ 8863.983504] Lustre: server umount lustre-MDT0000 complete [ 8867.582364] LustreError: 229826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396146 with bad export cookie 14296327804827207255 [ 8867.590893] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8867.594154] LustreError: 229826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8867.940447] Lustre: server umount lustre-MDT0001 complete [ 8886.080197] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8896.994242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8897.420681] LustreError: 234734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8897.447502] LustreError: 234734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8897.488274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8902.494627] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8902.633294] LustreError: 234735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8907.748107] LustreError: 234734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8911.179367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8911.533037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8915.924356] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8918.717353] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8925.475806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8925.887969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8932.398868] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8940.282516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8940.479386] Lustre: lustre-OST0001: new disk, initializing [ 8940.484390] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8940.491407] Lustre: 237044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8947.014909] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8948.325571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8948.333864] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8948.466694] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8957.261798] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8967.387969] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8979.753834] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 06:57:38 (1787396258) [ 8983.526911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8983.537071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8983.548027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8987.941550] Lustre: server umount lustre-MDT0000 complete [ 8989.668241] LustreError: 234735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8989.695133] LustreError: 234735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8991.442531] LustreError: 234715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396270 with bad export cookie 14296327804827208137 [ 8991.452634] LustreError: 234715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8991.458394] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8991.865204] Lustre: server umount lustre-MDT0001 complete [ 9005.545648] Lustre: server umount lustre-OST0000 complete [ 9019.687493] Lustre: server umount lustre-OST0001 complete [ 9027.109141] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 9036.423810] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9072.390123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9072.690197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9072.725269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9072.832546] Lustre: lustre-MDT0000: new disk, initializing [ 9072.964624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9072.977276] Lustre: Skipped 1 previous similar message [ 9073.002211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9078.988595] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9091.617722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9091.776400] Lustre: 240990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9091.830721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9091.834915] Lustre: Skipped 1 previous similar message [ 9091.975678] Lustre: lustre-MDT0001: new disk, initializing [ 9092.229637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9092.256538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9098.555698] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9108.085708] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9114.288865] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9120.962907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9121.231320] Lustre: lustre-OST0000: new disk, initializing [ 9121.234844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9121.242943] Lustre: 242963:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9121.361573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9121.365611] Lustre: Skipped 1 previous similar message [ 9122.651276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9122.676746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9122.781112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9128.280695] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9137.593531] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9142.435608] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9142.698031] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9147.423985] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9147.719884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9153.512208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9153.514749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9153.519514] LustreError: Skipped 1 previous similar message [ 9153.523455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9153.523462] Lustre: Skipped 4 previous similar messages [ 9153.549265] Lustre: Skipped 4 previous similar messages [ 9158.632224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9158.650387] Lustre: Skipped 2 previous similar messages [ 9164.255155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9164.485867] Lustre: server umount lustre-OST0000 complete [ 9173.988505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9173.994284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9174.034494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9174.064810] Lustre: Skipped 1 previous similar message [ 9174.102563] Lustre: Skipped 3 previous similar messages [ 9175.848729] Lustre: server umount lustre-MDT0000 complete [ 9179.106471] LustreError: 242810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9179.129264] LustreError: 242810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9180.286254] LustreError: 241785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396459 with bad export cookie 14296327804827210272 [ 9180.297826] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9180.308479] LustreError: 241785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9180.952961] Lustre: server umount lustre-MDT0001 complete [ 9190.582578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9190.959437] LustreError: 245192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9191.047060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9195.793243] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9205.612967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9212.025600] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9222.472796] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9229.928863] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9237.206035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9243.750494] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9251.038498] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9398.753449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9398.756836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9398.764334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9398.775544] Lustre: Skipped 1 previous similar message [ 9402.918065] Lustre: server umount lustre-OST0000 complete [ 9413.936224] Lustre: server umount lustre-MDT0000 complete [ 9414.112925] LustreError: 246132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9414.146094] LustreError: 246132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9418.354333] LustreError: 245173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396697 with bad export cookie 14296327804827211154 [ 9418.369266] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9418.374349] LustreError: 245173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9419.009894] Lustre: server umount lustre-MDT0001 complete [ 9520.901399] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 9523.400255] Key type lgssc unregistered [ 9523.714261] LNet: 248802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9523.718810] LNetError: 248802:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9523.729947] LNet: Removed LNI 192.168.201.105@tcp [ 9524.582437] Key type .llcrypt unregistered [ 9524.588422] Key type ._llcrypt unregistered [ 9551.230874] Key type ._llcrypt registered [ 9551.232952] Key type .llcrypt registered [ 9551.334856] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 9568.050899] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9569.217498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9569.468078] alg: No test for adler32 (adler32-zlib) [ 9570.717695] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 9571.045592] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 9572.775173] Key type lgssc registered [ 9574.279904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9621.717589] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9633.095992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9633.146267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9634.546281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9634.595632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9634.737896] Lustre: lustre-MDT0000: new disk, initializing [ 9634.878506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9634.977782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9640.555724] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9651.742190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9651.888871] Lustre: 252878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9652.036444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9652.051188] Lustre: Skipped 1 previous similar message [ 9652.204373] Lustre: lustre-MDT0001: new disk, initializing [ 9652.322969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9652.379832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9652.396808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9656.315126] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9661.539829] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9669.180860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9669.445356] Lustre: lustre-OST0000: new disk, initializing [ 9669.447501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9669.451679] Lustre: 254508:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9669.534252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9675.903945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9675.919719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9676.069204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9676.872199] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9688.769883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9689.019423] Lustre: lustre-OST0001: new disk, initializing [ 9689.022487] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9689.028326] Lustre: 255380:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9689.136089] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9694.816524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9694.829792] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9695.021377] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9696.578715] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9708.041500] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9714.316390] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9728.994774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9729.012848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9729.037738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9731.047599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9731.050028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9731.096648] Lustre: Skipped 3 previous similar messages [ 9734.143983] Lustre: server umount lustre-MDT0000 complete [ 9737.614953] LustreError: 252871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397016 with bad export cookie 12191814826411107540 [ 9737.623649] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9737.933570] Lustre: server umount lustre-MDT0001 complete [ 9751.368898] Lustre: server umount lustre-OST0000 complete [ 9765.856055] Lustre: server umount lustre-OST0001 complete [ 9782.847097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9793.002851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9801.721548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9811.891721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9834.892841] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9846.642972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9846.671419] Lustre: lustre-MDT0000: reset Object Index mappings [ 9847.397656] LustreError: 260391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9847.411807] LustreError: 260391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9847.496070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9852.896591] LustreError: 260392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9853.433354] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9857.005381] LustreError: 260391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9862.120749] LustreError: 260392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9863.004875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9863.033030] Lustre: lustre-MDT0001: reset Object Index mappings [ 9863.632601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9868.635470] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9872.102885] Lustre: 261534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9878.149364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9886.058317] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9888.940088] LustreError: 261889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9892.029069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9895.384792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9895.753465] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9895.762366] Lustre: Skipped 1 previous similar message [ 9901.064113] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9902.722843] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9910.857935] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9914.799235] Lustre: 263407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9919.970356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9919.983616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9920.007045] Lustre: Skipped 2 previous similar messages [ 9920.022458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9921.789646] Lustre: server umount lustre-MDT0000 complete [ 9926.090330] LustreError: 261914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397205 with bad export cookie 12191814826411114330 [ 9926.093214] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9926.107513] LustreError: 261914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9926.379823] Lustre: server umount lustre-MDT0001 complete [ 9940.894056] Lustre: server umount lustre-OST0000 complete [ 9942.825914] Lustre: 250383:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397205/real 1787397205] req@ffff8f0d483e0a80 x1874221439512960/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787397221 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9942.856728] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9942.872657] Lustre: Skipped 3 previous similar messages [ 9944.967505] Lustre: server umount lustre-OST0001 complete [ 9965.036638] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:14:03 (1787397243) [ 9981.617501] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 9991.807107] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10037.854497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10038.104176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10038.136223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10038.250594] Lustre: lustre-MDT0000: new disk, initializing [10038.347379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10038.371451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10043.592205] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10055.530933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10055.719701] Lustre: 267660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10055.823641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10055.828604] Lustre: Skipped 1 previous similar message [10055.936608] Lustre: lustre-MDT0001: new disk, initializing [10056.091050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10056.107895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10060.724400] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10070.233717] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10076.219461] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10083.664685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10083.882575] Lustre: lustre-OST0000: new disk, initializing [10083.885890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10083.890840] Lustre: 269632:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10085.225823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10085.240753] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10085.381133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10090.898306] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10099.499654] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10103.881113] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10104.142627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10111.629424] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10112.121557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10116.068702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10116.124980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10116.142215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10116.145864] Lustre: Skipped 1 previous similar message [10116.174904] Lustre: Skipped 5 previous similar messages [10120.983349] Lustre: server umount lustre-OST0000 complete [10127.841859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10127.852063] LustreError: Skipped 1 previous similar message [10127.856498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10127.886153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10133.091872] Lustre: server umount lustre-MDT0000 complete [10136.545879] LustreError: 269135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10136.562966] LustreError: 269135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10137.729135] LustreError: 267651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397416 with bad export cookie 12191814826411116416 [10137.730526] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10137.744178] LustreError: 267651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10138.504796] Lustre: server umount lustre-MDT0001 complete [10157.954723] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10168.522334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10168.912878] LustreError: 272565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10168.994042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10168.999721] Lustre: Skipped 2 previous similar messages [10173.287632] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10181.698857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10187.187700] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10190.092774] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10197.428777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10204.946562] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10215.572680] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10215.704775] Lustre: lustre-OST0001: new disk, initializing [10215.706791] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10215.709863] Lustre: 274874:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10217.541067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10217.557649] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10217.719949] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10222.543831] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10232.231091] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10235.816713] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10240.876724] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10241.059464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10248.677887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10248.690216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10248.701053] Lustre: Skipped 1 previous similar message [10248.701801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10248.701806] Lustre: Skipped 2 previous similar messages [10254.280205] Lustre: server umount lustre-MDT0000 complete [10257.956491] LustreError: 272546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397536 with bad export cookie 12191814826411117305 [10257.961606] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10257.973879] LustreError: 272546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10258.601435] Lustre: server umount lustre-MDT0001 complete [10273.082476] Lustre: server umount lustre-OST0000 complete [10274.277648] Lustre: 250383:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397537/real 1787397537] req@ffff8f0d44535880 x1874221439622016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787397553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10277.384247] Lustre: server umount lustre-OST0001 complete [10296.459934] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10306.716828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10307.171255] LustreError: 277930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10307.186419] LustreError: 277930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10307.245910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10307.253170] Lustre: Skipped 3 previous similar messages [10311.983885] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10321.289287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10326.967596] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10330.301718] Lustre: 279070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10336.478994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10342.344177] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10352.356513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10352.734434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10358.299604] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10360.732662] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10368.279820] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10372.392774] Lustre: 280948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10377.593748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10377.857969] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10387.890925] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:21:06 (1787397666) [10403.808331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10403.817771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10403.846496] Lustre: Skipped 4 previous similar messages [10403.857439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10403.867413] Lustre: Skipped 7 previous similar messages [10408.826806] Lustre: server umount lustre-MDT0000 complete [10409.445151] LustreError: 277926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10409.472439] LustreError: 277926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10412.735135] LustreError: 277910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397691 with bad export cookie 12191814826411123843 [10412.747216] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10412.752069] LustreError: 277910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10427.359191] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10427.733713] Lustre: server umount lustre-MDT0001 complete [10438.674834] Lustre: server umount lustre-OST0000 complete [10442.670883] Lustre: server umount lustre-OST0001 complete [10451.280364] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [10463.401828] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10511.713377] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10520.134660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10520.265836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10520.287412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10520.357866] Lustre: lustre-MDT0000: new disk, initializing [10520.441982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10523.781467] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10533.611294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10533.820774] Lustre: 285186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10534.259355] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10540.570947] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10551.665570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10551.840410] Lustre: 286819:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10553.531414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10558.718733] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10569.247558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10569.372784] Lustre: 287691:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10569.496220] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10569.501215] Lustre: Skipped 6 previous similar messages [10570.619153] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10570.635172] Lustre: Skipped 1 previous similar message [10570.811747] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10576.331950] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10586.498655] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10590.386611] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10590.391854] Lustre: Skipped 1 previous similar message [10606.789142] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:24:45 (1787397885) [10611.169445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10611.174745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10611.182381] Lustre: Skipped 5 previous similar messages [10611.186870] LustreError: 288465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10611.194720] LustreError: 288465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10611.249584] Lustre: server umount lustre-MDT0000 complete [10615.376555] LustreError: 285180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397894 with bad export cookie 12191814826411131473 [10615.392428] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10615.899316] Lustre: server umount lustre-MDT0001 complete [10630.620437] Lustre: server umount lustre-OST0000 complete [10632.671704] Lustre: 250384:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397895/real 1787397895] req@ffff8f0e47489500 x1874221439768960/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787397911 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10635.415932] Lustre: server umount lustre-OST0001 complete [10644.461828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10649.971953] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10658.788303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10664.638873] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10672.911364] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10678.547660] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10686.201482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10693.155183] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10700.705528] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10719.861852] loop1: detected capacity change from 0 to 204800000 [10720.245323] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10720.273075] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10720.303514] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10726.781467] loop1: detected capacity change from 0 to 204800000 [10731.660332] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10731.708787] LustreError: 289924:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs21112: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10731.714898] LustreError: 289924:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs21112 [10731.717097] LustreError: 289924:0:(mgs_handler.c:517:mgs_target_reg()) fs21112-MDT0001: Error checking target index: rc = -13 [10731.720670] LustreError: fs21112-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10731.725228] LustreError: 293657:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10731.729493] LustreError: 293657:0:(tgt_mount.c:2129:server_put_super()) no obd fs21112-MDT0001 [10731.732335] LustreError: 293657:0:(tgt_mount.c:116:server_deregister_mount()) fs21112-MDT0001 not registered [10731.765342] Lustre: server umount fs21112-MDT0001 complete [10731.771061] LustreError: 293657:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10739.582279] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10739.671805] Lustre: Setting parameter fs21112-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs21112-MDT0001 [10739.718236] Lustre: srv-fs21112-MDT0001: No data found on store. Initialize space. [10739.722639] Lustre: Skipped 4 previous similar messages [10739.791756] Lustre: fs21112-MDT0001: new disk, initializing [10739.795443] Lustre: Skipped 3 previous similar messages [10739.919477] Lustre: srv-fs21112-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10739.922191] Lustre: 294123:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21112-MDT0001-mdtlov: can't create new llog: rc = -115 [10739.942496] LustreError: 294123:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21112-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10743.590765] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10750.637335] Lustre: server umount fs21112-MDT0001 complete [10764.895633] loop1: detected capacity change from 0 to 204800000 [10770.167756] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10770.595545] Lustre: 295689:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21112-MDT0001-mdtlov: can't create new llog: rc = -115 [10770.608315] LustreError: 295689:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21112-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10770.914734] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10775.087937] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10779.726580] Lustre: server umount fs21112-MDT0001 complete [10788.614076] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [10788.620288] Lustre: Skipped 10 previous similar messages [10791.394599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10791.409840] Lustre: Skipped 5 previous similar messages [10795.004440] Lustre: server umount lustre-MDT0000 complete [10803.360282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10803.480971] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10808.453562] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10808.816739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10815.647297] loop1: detected capacity change from 0 to 204800000 [10820.705880] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10820.995995] Lustre: 298135:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21112-MDT0001-mdtlov: can't create new llog: rc = -115 [10821.009375] LustreError: 298135:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21112-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10824.960701] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10829.924541] Lustre: server umount fs21112-MDT0001 complete [10849.760469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10849.768194] LustreError: Skipped 1 previous similar message [10854.216273] Lustre: server umount lustre-OST0000 complete [10860.001020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10860.011865] Lustre: Skipped 7 previous similar messages [10873.311283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10873.788797] Lustre: server umount lustre-MDT0000 complete [10874.336184] LustreError: 289936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.370228] LustreError: 289936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 104 previous similar messages [10877.662052] LustreError: 290679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398156 with bad export cookie 12191814826411134154 [10877.673209] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10877.675252] LustreError: 290679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10878.017917] Lustre: server umount lustre-MDT0001 complete [10888.368857] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10890.992671] Key type lgssc unregistered [10891.297932] LNet: 300480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10891.308686] LNetError: 300480:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10891.330349] LNet: Removed LNI 192.168.201.105@tcp [10892.007360] Key type .llcrypt unregistered [10892.009767] Key type ._llcrypt unregistered [10909.812901] Key type ._llcrypt registered [10909.823962] Key type .llcrypt registered [10909.942657] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10910.971519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10911.000750] alg: No test for adler32 (adler32-zlib) [10912.180264] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [10912.410217] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10914.143729] Key type lgssc registered [10915.334534] Lustre: Echo OBD driver; http://www.lustre.org/ [10923.164058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10923.180077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10924.662141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10930.316696] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10934.241154] LustreError: 301751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10934.273864] LustreError: 301751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10938.797439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10939.296905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10944.707945] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10951.839725] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10957.826778] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10964.159372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10964.484345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10965.498214] LustreError: 303581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10969.608955] LustreError: 303582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10969.643160] LustreError: 303582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10970.918626] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10974.692535] LustreError: 303581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10974.709666] LustreError: 303581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10978.752808] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10979.810791] LustreError: 303582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10979.828527] LustreError: 303582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10990.061860] LustreError: 304254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10990.096039] LustreError: 304254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10999.489819] loop1: detected capacity change from 0 to 204800000 [10999.799072] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10999.823434] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10999.839380] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11005.914154] loop1: detected capacity change from 0 to 204800000 [11007.215255] LustreError: 304254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11007.241599] LustreError: 304254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [11011.908432] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11012.063967] Lustre: fs20465-OST0000: new disk, initializing [11012.070861] Lustre: srv-fs20465-OST0000: No data found on store. Initialize space. [11012.077200] Lustre: 305490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11012.205569] Lustre: fs20465-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11018.974571] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11028.152385] Lustre: server umount fs20465-OST0000 complete [11039.570083] loop1: detected capacity change from 0 to 204800000 [11041.250488] LustreError: 303582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11041.292275] LustreError: 303582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [11045.939248] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11046.540915] Lustre: fs20465-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11051.500991] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11053.195746] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11062.225059] loop2: detected capacity change from 0 to 204800000 [11062.578309] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11062.599533] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11062.637802] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11068.419322] loop2: detected capacity change from 0 to 204800000 [11074.555744] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11074.677948] LustreError: fs20465-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11074.688196] LustreError: fs20465-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11074.698150] LustreError: 308528:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11074.702890] LustreError: 308528:0:(tgt_mount.c:2129:server_put_super()) no obd fs20465-OST0001 [11074.714775] LustreError: 308528:0:(tgt_mount.c:116:server_deregister_mount()) fs20465-OST0001 not registered [11074.757432] Lustre: server umount fs20465-OST0001 complete [11074.763575] LustreError: 308528:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11085.097442] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11085.701440] Lustre: fs20465-OST0001: new disk, initializing [11085.718624] Lustre: srv-fs20465-OST0001: No data found on store. Initialize space. [11085.748626] Lustre: 309079:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11086.016962] Lustre: fs20465-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11092.711154] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11102.143656] Lustre: server umount fs20465-OST0000 complete [11107.819623] LustreError: 303581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11107.851140] LustreError: 303581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11111.603680] Lustre: server umount fs20465-OST0001 complete [11124.640750] loop1: detected capacity change from 0 to 204800000 [11130.980175] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11131.304590] Lustre: fs20465-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11138.075594] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11145.568291] loop2: detected capacity change from 0 to 204800000 [11151.285149] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11158.158839] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11167.835393] Lustre: server umount fs20465-OST0000 complete [11177.843388] Lustre: server umount fs20465-OST0001 complete [11194.869519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11194.869939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11194.882315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11194.887672] Lustre: Skipped 1 previous similar message [11194.910190] LustreError: Skipped 1 previous similar message [11199.974068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11199.982209] Lustre: Skipped 2 previous similar messages [11205.087209] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11205.093536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11205.094350] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11205.121017] Lustre: Skipped 1 previous similar message [11205.286268] Lustre: server umount lustre-OST0000 complete [11210.222667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11210.244388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11210.250488] Lustre: Skipped 1 previous similar message [11215.335737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11215.656466] Lustre: server umount lustre-MDT0000 complete [11219.491269] LustreError: 301733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398498 with bad export cookie 10668339321954889025 [11219.518897] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11219.796882] Lustre: server umount lustre-MDT0001 complete [11231.292186] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11234.852865] Key type lgssc unregistered [11235.131496] LNet: 315004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11235.134371] LNetError: 315004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11235.143656] LNet: Removed LNI 192.168.201.105@tcp [11236.088518] Key type .llcrypt unregistered [11236.091967] Key type ._llcrypt unregistered [11270.531463] Key type ._llcrypt registered [11270.534357] Key type .llcrypt registered [11270.653155] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [11288.894446] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11290.144323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11290.290639] alg: No test for adler32 (adler32-zlib) [11291.315888] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [11291.561095] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11293.176251] Key type lgssc registered [11294.374178] Lustre: Echo OBD driver; http://www.lustre.org/ [11335.704624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11335.732861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11337.113504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11337.170737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11337.319834] Lustre: lustre-MDT0000: new disk, initializing [11337.458583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11337.481371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11343.519838] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11355.662745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11355.839168] Lustre: 318867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11355.910449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11355.913874] Lustre: Skipped 1 previous similar message [11355.996850] Lustre: lustre-MDT0001: new disk, initializing [11356.096174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11356.158435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11356.178096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11361.517351] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11370.825933] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11377.498687] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11384.672064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11384.855111] Lustre: lustre-OST0000: new disk, initializing [11384.866839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11384.870322] Lustre: 320835:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11384.925527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11392.594437] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11393.650126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11393.667816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11393.806579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11404.100306] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11406.265367] Lustre: Mounted lustre-client [11407.462051] Lustre: Unmounted lustre-client [11419.627818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11419.633315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11419.648985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11419.656967] Lustre: Skipped 1 previous similar message [11421.672338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11421.675631] Lustre: Skipped 1 previous similar message [11426.791240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11426.798435] Lustre: Skipped 1 previous similar message [11431.392181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11431.700453] Lustre: server umount lustre-OST0000 complete [11437.026238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11437.030728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11437.047075] Lustre: Skipped 2 previous similar messages [11437.062576] Lustre: Skipped 1 previous similar message [11442.101147] Lustre: server umount lustre-MDT0000 complete [11446.077675] LustreError: 318859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398725 with bad export cookie 4689651069917922994 [11446.088115] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11446.092183] LustreError: 318859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11446.561848] Lustre: server umount lustre-MDT0001 complete [11458.504996] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11461.407855] Key type lgssc unregistered [11461.535202] LNet: 1 local NIs in recovery (showing 1): 192.168.201.105@tcp [11461.690733] LNet: 322996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11461.708750] LNetError: 322996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11535.474283] LNet: Removed LNI 192.168.201.105@tcp [11536.441739] Key type .llcrypt unregistered [11536.443808] Key type ._llcrypt unregistered [11556.051316] Key type ._llcrypt registered [11556.056977] Key type .llcrypt registered [11556.174917] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [11574.627394] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11575.909597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11575.962949] alg: No test for adler32 (adler32-zlib) [11577.121788] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [11577.346473] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11578.992119] Key type lgssc registered [11580.508027] Lustre: Echo OBD driver; http://www.lustre.org/ [11613.902506] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:41:32 (1787398892) [11621.145977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11621.176342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11622.573504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11622.627392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11622.860850] Lustre: lustre-MDT0000: new disk, initializing [11623.016545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11623.056692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11628.526227] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11641.275559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11641.430228] Lustre: 326219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11641.468853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11641.472536] Lustre: Skipped 1 previous similar message [11641.528408] Lustre: lustre-MDT0001: new disk, initializing [11641.583496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11641.609713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11641.622907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11646.973439] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11657.928428] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11664.760480] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11672.909513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11673.276688] Lustre: lustre-OST0000: new disk, initializing [11673.284558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11673.294641] Lustre: 328186:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11673.404758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11679.302106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11679.317979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11679.441973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11680.710455] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11691.339920] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11710.434229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11710.436108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11710.456522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11710.471567] Lustre: Skipped 1 previous similar message [11710.489559] Lustre: Skipped 1 previous similar message [11712.314735] Lustre: server umount lustre-OST0000 complete [11718.628242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11718.637430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11718.654829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11720.674636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11720.682469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11720.700444] Lustre: Skipped 1 previous similar message [11722.704122] Lustre: server umount lustre-MDT0000 complete [11726.766806] LustreError: 326210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399005 with bad export cookie 10515190412806326059 [11726.786205] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11727.182263] Lustre: server umount lustre-MDT0001 complete [11739.401123] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11742.455865] Key type lgssc unregistered [11742.795096] LNet: 330095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11742.824921] LNetError: 330095:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11742.867362] LNet: Removed LNI 192.168.201.105@tcp [11744.256451] Key type .llcrypt unregistered [11744.263289] Key type ._llcrypt unregistered [11769.458836] Key type ._llcrypt registered [11769.460644] Key type .llcrypt registered [11769.621496] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11770.799902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11770.892621] alg: No test for adler32 (adler32-zlib) [11771.946164] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [11772.236928] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11773.943241] Key type lgssc registered [11775.296821] Lustre: Echo OBD driver; http://www.lustre.org/ [11783.411571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11783.424729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11785.025527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11790.984835] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11794.428955] LustreError: 331369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11794.476256] LustreError: 331369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11799.543909] LustreError: 331370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11802.603388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11803.156971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11808.877914] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11816.470753] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11823.992649] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11831.884554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11832.140473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11838.643751] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11846.766857] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11866.234951] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11868.485538] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11874.785594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11874.801829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11874.830084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11878.166718] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [11878.177790] Lustre: Skipped 3 previous similar messages [11880.203939] Lustre: server umount lustre-MDT0000 complete [11880.429554] LustreError: 331370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11880.467680] LustreError: 331370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11883.234345] LustreError: 331365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11884.497217] LustreError: 331351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399163 with bad export cookie 9212706880933486086 [11884.512137] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11885.543297] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11885.561633] Lustre: Skipped 2 previous similar messages [11885.566804] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11888.350945] LustreError: 331365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11888.359295] LustreError: 331365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11888.402696] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [11891.356091] Lustre: server umount lustre-MDT0001 complete [11897.004425] Lustre: server umount lustre-OST0000 complete [11906.450415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11913.538688] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11918.472073] Lustre: 335574:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.105@tcp [11921.907137] Lustre: server umount MGS complete [11931.669856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11932.044756] LustreError: 336145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11932.053189] LustreError: 336145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11932.134363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11936.558574] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11945.108592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11945.625366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11951.390282] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11961.448875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11962.017846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11969.308269] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11977.214595] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11979.336221] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11985.570866] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11987.463512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11993.075615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11993.077173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11993.081549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11993.081560] Lustre: Skipped 1 previous similar message [11993.089963] Lustre: Skipped 1 previous similar message [11994.587887] Lustre: server umount lustre-OST0000 complete [12001.760491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12001.780931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12001.803032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12001.815287] Lustre: Skipped 1 previous similar message [12005.207147] Lustre: server umount lustre-MDT0000 complete [12008.421172] LustreError: 336164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12008.444399] LustreError: 336164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12009.538549] LustreError: 336127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399288 with bad export cookie 9212706880933486919 [12009.544045] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12009.557056] LustreError: 336127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12009.964527] Lustre: server umount lustre-MDT0001 complete [12023.695804] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12026.945644] Key type lgssc unregistered [12027.383803] LNet: 339582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12027.402748] LNetError: 339582:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12027.432848] LNet: Removed LNI 192.168.201.105@tcp [12028.349982] Key type .llcrypt unregistered [12028.354401] Key type ._llcrypt unregistered [12047.656280] Key type ._llcrypt registered [12047.657516] Key type .llcrypt registered [12047.761852] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [12065.470978] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12066.536474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12066.966858] alg: No test for adler32 (adler32-zlib) [12068.170335] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [12068.533723] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [12070.231370] Key type lgssc registered [12071.683797] Lustre: Echo OBD driver; http://www.lustre.org/ [12106.386836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12106.403471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12107.649532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12107.677685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12107.822674] Lustre: lustre-MDT0000: new disk, initializing [12107.937658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12107.962853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12112.380684] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12122.612506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12122.735786] Lustre: 342750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12122.782878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12122.791091] Lustre: Skipped 1 previous similar message [12122.882622] Lustre: lustre-MDT0001: new disk, initializing [12122.977716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12123.017413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12123.030941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12127.396269] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12136.858915] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12142.581287] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12149.284798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12149.600467] Lustre: lustre-OST0000: new disk, initializing [12149.605574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12149.618949] Lustre: 344719:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12149.707878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12155.141922] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12159.056282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12159.072319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12159.196405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12163.498286] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12167.634094] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12167.836665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12172.352839] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12172.526423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12174.821361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12174.825164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12174.828930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12174.841247] Lustre: Skipped 1 previous similar message [12179.949996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12179.952191] Lustre: Skipped 2 previous similar messages [12180.662317] Lustre: server umount lustre-OST0000 complete [12184.544903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12184.554767] LustreError: Skipped 1 previous similar message [12184.557055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12184.585200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12190.176890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12190.185304] Lustre: Skipped 3 previous similar messages [12190.996365] Lustre: server umount lustre-MDT0000 complete [12194.594659] LustreError: 343544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399473 with bad export cookie 13265241096194495571 [12194.601507] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12194.605450] LustreError: 343544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12195.323052] Lustre: server umount lustre-MDT0001 complete [12205.177857] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 07:51:23 (1787399483) [12221.695975] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [12231.454968] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12267.548699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12267.850439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12267.906372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12268.023770] Lustre: lustre-MDT0000: new disk, initializing [12268.120844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12268.135556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12272.590353] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12283.179100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12283.253583] Lustre: 349478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12283.323085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12283.331853] Lustre: Skipped 1 previous similar message [12283.332466] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12283.401485] Lustre: lustre-MDT0001: new disk, initializing [12283.485991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12283.530700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12283.539483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12287.604826] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12298.142212] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12306.014760] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12314.005298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12314.378834] Lustre: lustre-OST0000: new disk, initializing [12314.386690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12314.395855] Lustre: 351447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12314.491453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12320.863499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12320.871315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12321.067326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12321.358690] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12331.756148] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12367.331857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12367.334458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12367.340220] LustreError: Skipped 1 previous similar message [12367.343445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12367.365382] Lustre: Skipped 2 previous similar messages [12371.416904] Lustre: server umount lustre-OST0000 complete [12375.520432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12375.538479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12381.520420] Lustre: server umount lustre-MDT0000 complete [12385.640045] LustreError: 349471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399664 with bad export cookie 13265241096194496446 [12385.653113] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12385.672066] LustreError: 349471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12386.105615] Lustre: server umount lustre-MDT0001 complete [12399.488610] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12402.637591] Key type lgssc unregistered [12402.988398] LNet: 353622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12402.995866] LNetError: 353622:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12403.015472] LNet: Removed LNI 192.168.201.105@tcp [12404.325396] Key type .llcrypt unregistered [12404.327937] Key type ._llcrypt unregistered [12505.388535] Key type ._llcrypt registered [12505.390711] Key type .llcrypt registered [12505.495273] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12505.849251] Key type .llcrypt unregistered [12505.850576] Key type ._llcrypt unregistered [12581.275990] Key type ._llcrypt registered [12581.277536] Key type .llcrypt registered [12581.465773] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12581.944691] Key type .llcrypt unregistered [12581.954103] Key type ._llcrypt unregistered [12619.239521] Key type ._llcrypt registered [12619.242361] Key type .llcrypt registered [12619.305105] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12619.750723] Key type .llcrypt unregistered [12619.752076] Key type ._llcrypt unregistered [12653.022551] Key type ._llcrypt registered [12653.024762] Key type .llcrypt registered [12653.185283] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12653.494623] Key type .llcrypt unregistered [12653.497165] Key type ._llcrypt unregistered [12736.520986] Key type ._llcrypt registered [12736.523269] Key type .llcrypt registered [12736.638766] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12736.985588] Key type .llcrypt unregistered [12736.987680] Key type ._llcrypt unregistered [12768.489762] Key type ._llcrypt registered [12768.492514] Key type .llcrypt registered [12768.641161] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12768.932498] Key type .llcrypt unregistered [12768.934469] Key type ._llcrypt unregistered [12803.055230] Key type ._llcrypt registered [12803.056753] Key type .llcrypt registered [12803.219517] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12803.688702] Key type .llcrypt unregistered [12803.691158] Key type ._llcrypt unregistered [12838.764413] Key type ._llcrypt registered [12838.766361] Key type .llcrypt registered [12838.880391] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12839.177415] Key type .llcrypt unregistered [12839.182366] Key type ._llcrypt unregistered [12872.539843] Key type ._llcrypt registered [12872.542187] Key type .llcrypt registered [12872.627311] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12872.884789] Key type .llcrypt unregistered [12872.888741] Key type ._llcrypt unregistered [12904.414157] Key type ._llcrypt registered [12904.415978] Key type .llcrypt registered [12904.530667] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12904.837578] Key type .llcrypt unregistered [12904.843093] Key type ._llcrypt unregistered [12961.781255] Key type ._llcrypt registered [12961.783883] Key type .llcrypt registered [12961.919385] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12962.265425] Key type .llcrypt unregistered [12962.267227] Key type ._llcrypt unregistered [12994.746162] Key type ._llcrypt registered [12994.748323] Key type .llcrypt registered [12994.886541] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12995.245775] Key type .llcrypt unregistered [12995.248307] Key type ._llcrypt unregistered [13025.989801] Key type ._llcrypt registered [13025.994104] Key type .llcrypt registered [13026.077507] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13026.377444] Key type .llcrypt unregistered [13026.379641] Key type ._llcrypt unregistered [13083.030958] Key type ._llcrypt registered [13083.035400] Key type .llcrypt registered [13083.135182] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [13099.985855] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [13100.784384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13100.854350] alg: No test for adler32 (adler32-zlib) [13101.863214] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [13102.172554] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [13103.855178] Key type lgssc registered [13105.084714] Lustre: Echo OBD driver; http://www.lustre.org/ [13139.767514] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13139.787377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13141.108915] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13141.134304] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13141.226464] Lustre: tst250fs-MDT0000: new disk, initializing [13141.289933] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13141.301301] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13146.556506] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13159.212736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13159.278189] Lustre: 373987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13159.304583] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13159.306575] Lustre: Skipped 1 previous similar message [13159.343568] Lustre: tst250fs-MDT0001: new disk, initializing [13159.387134] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13159.416856] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13159.428169] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13164.123986] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13169.014577] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13175.712838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13175.956821] Lustre: tst250fs-OST0000: new disk, initializing [13175.961563] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13175.970106] Lustre: 375621:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13176.043248] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13179.418217] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13179.428852] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13179.583874] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13181.942368] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13193.892539] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13194.375172] Lustre: tst250fs-OST0001: new disk, initializing [13194.385974] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13194.393436] Lustre: 376491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13194.689086] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13202.273210] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13204.067911] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13204.074165] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13204.143168] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13208.395294] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13211.548684] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13211.551557] Lustre: Skipped 2 previous similar messages [13235.171298] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13235.194134] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13235.207490] Lustre: Skipped 1 previous similar message [13235.233490] Lustre: Skipped 1 previous similar message [13237.757460] Lustre: server umount tst250fs-MDT0000 complete [13242.088411] LustreError: 373979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400521 with bad export cookie 8882266063972975140 [13242.103202] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13242.114688] LustreError: 373979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13242.752401] Lustre: server umount tst250fs-MDT0001 complete [13257.408784] Lustre: server umount tst250fs-OST0000 complete [13261.984807] Lustre: 372211:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787400524/real 1787400524] req@ffff8f0e7188b480 x1874225141979776/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787400540 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13262.012754] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13262.028502] Lustre: Skipped 2 previous similar messages [13262.882677] Lustre: server umount tst250fs-OST0001 complete [13283.951957] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [13293.219898] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [13325.426347] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:10:04 (1787400604) [13326.852298] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13342.236505] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [13352.006511] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [13385.208147] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13103 sec ========= 08:11:03 (1787400663) [13387.039796] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:11:05 (1787400665) === [13391.006767] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:11:09 (1787400669) === [13437.015391] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13439.807422] Key type lgssc unregistered [13440.095879] LNet: 385644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13440.101602] LNetError: 385644:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13440.113970] LNet: Removed LNI 192.168.201.105@tcp [13441.058189] Key type .llcrypt unregistered [13441.060409] Key type ._llcrypt unregistered