[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479759970 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002262] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008013] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010099] LSM: Security Framework initializing [ 0.011032] Yama: becoming mindful. [ 0.012024] SELinux: Initializing. [ 0.013044] *** VALIDATE selinux *** [ 0.020417] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024267] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027095] *** VALIDATE tmpfs *** [ 0.028358] *** VALIDATE proc *** [ 0.029169] *** VALIDATE cgroup *** [ 0.030007] *** VALIDATE cgroup2 *** [ 0.031217] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034023] Spectre V2 : User space: Vulnerable [ 0.035004] Speculative Store Bypass: Vulnerable [ 0.038396] debug: unmapping init [mem 0xffffffff90459000-0xffffffff90460fff] [ 0.040189] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041717] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042033] ... version: 2 [ 0.043017] ... bit width: 48 [ 0.044010] ... generic registers: 4 [ 0.045010] ... value mask: 0000ffffffffffff [ 0.046020] ... max period: 00007fffffffffff [ 0.047015] ... fixed-purpose events: 3 [ 0.048015] ... event mask: 000000070000000f [ 0.049312] rcu: Hierarchical SRCU implementation. [ 0.051470] smp: Bringing up secondary CPUs ... [ 0.052620] x86: Booting SMP configuration: [ 0.053031] .... node #0, CPUs: #1 #2 #3 [ 0.056174] smp: Brought up 1 node, 4 CPUs [ 0.057895] smpboot: Max logical packages: 1 [ 0.058013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.126470] node 0 deferred pages initialised in 66ms [ 0.129117] devtmpfs: initialized [ 0.130151] x86/mm: Memory block size: 128MB [ 0.132141] gcov: version magic: 0x41383552 [ 0.134035] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.135096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.136275] pinctrl core: initialized pinctrl subsystem [ 0.137156] [ 0.137480] ************************************************************* [ 0.138014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.139015] ** ** [ 0.140019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.141014] ** ** [ 0.142018] ** This means that this kernel is built to expose internal ** [ 0.143018] ** IOMMU data structures, which may compromise security on ** [ 0.144013] ** your system. ** [ 0.145020] ** ** [ 0.146013] ** If you see this message and you are not debugging the ** [ 0.147014] ** kernel, report this immediately to your vendor! ** [ 0.148013] ** ** [ 0.149014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150015] ************************************************************* [ 0.151568] NET: Registered protocol family 16 [ 0.152489] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.153066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.154072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.156012] cpuidle: using governor menu [ 0.157655] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.161505] PCI: Using configuration type 1 for base access [ 0.163144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.174050] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.175022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.176172] cryptd: max_cpu_qlen set to 1000 [ 0.177227] ACPI: Added _OSI(Module Device) [ 0.178030] ACPI: Added _OSI(Processor Device) [ 0.179017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.180014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.184146] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.187167] ACPI: Interpreter enabled [ 0.188064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.189015] ACPI: Using IOAPIC for interrupt routing [ 0.190134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.191305] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.199047] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.200033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.201023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.202106] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.204288] acpiphp: Slot [2] registered [ 0.205128] acpiphp: Slot [5] registered [ 0.206115] acpiphp: Slot [6] registered [ 0.207092] acpiphp: Slot [7] registered [ 0.208072] acpiphp: Slot [8] registered [ 0.209066] acpiphp: Slot [9] registered [ 0.210157] acpiphp: Slot [10] registered [ 0.211078] acpiphp: Slot [3] registered [ 0.211973] acpiphp: Slot [4] registered [ 0.212083] acpiphp: Slot [11] registered [ 0.213055] acpiphp: Slot [12] registered [ 0.214083] acpiphp: Slot [13] registered [ 0.215038] acpiphp: Slot [14] registered [ 0.216067] acpiphp: Slot [15] registered [ 0.217075] acpiphp: Slot [16] registered [ 0.218107] acpiphp: Slot [17] registered [ 0.219107] acpiphp: Slot [18] registered [ 0.220097] acpiphp: Slot [19] registered [ 0.221108] acpiphp: Slot [20] registered [ 0.222100] acpiphp: Slot [21] registered [ 0.223097] acpiphp: Slot [22] registered [ 0.224065] acpiphp: Slot [23] registered [ 0.225070] acpiphp: Slot [24] registered [ 0.226034] acpiphp: Slot [25] registered [ 0.227134] acpiphp: Slot [26] registered [ 0.228080] acpiphp: Slot [27] registered [ 0.229095] acpiphp: Slot [28] registered [ 0.230098] acpiphp: Slot [29] registered [ 0.231083] acpiphp: Slot [30] registered [ 0.232082] acpiphp: Slot [31] registered [ 0.233042] PCI host bridge to bus 0000:00 [ 0.234017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.235022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.236022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.237018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.238020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.239019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.240134] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.241951] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.243310] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.249015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.252044] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.253017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.254021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.255015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.256580] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.257680] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.258044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.259766] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.262014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.270806] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.272015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.276058] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.279018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.282017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.290014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.295356] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.305018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.312031] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.332019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.343000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.355017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.364015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.392018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.402721] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.407018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.414019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.430018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.439086] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.446015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.452013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.465018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.472411] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.480025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.487015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.497016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.508393] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.511386] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.514360] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.516516] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.519211] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.526035] iommu: Default domain type: Passthrough [ 0.527419] SCSI subsystem initialized [ 0.529138] ACPI: bus type USB registered [ 0.530061] usbcore: registered new interface driver usbfs [ 0.531073] usbcore: registered new interface driver hub [ 0.533090] usbcore: registered new device driver usb [ 0.534160] pps_core: LinuxPPS API ver. 1 registered [ 0.536011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.538056] PTP clock support registered [ 0.540082] EDAC MC: Ver: 3.0.0 [ 0.541371] PCI: Using ACPI for IRQ routing [ 0.543891] NetLabel: Initializing [ 0.545012] NetLabel: domain hash size = 128 [ 0.547011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.549077] NetLabel: unlabeled traffic allowed by default [ 0.551141] vgaarb: loaded [ 0.553072] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.554011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.562495] clocksource: Switched to clocksource kvm-clock [ 0.670568] VFS: Disk quotas dquot_6.6.0 [ 0.672225] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.674860] *** VALIDATE ramfs *** [ 0.676162] *** VALIDATE hugetlbfs *** [ 0.677635] pnp: PnP ACPI init [ 0.680239] pnp: PnP ACPI: found 6 devices [ 0.702452] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.705662] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.707562] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.709859] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.712235] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.714670] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.717457] NET: Registered protocol family 2 [ 0.719629] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.723446] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.727165] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.732655] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.736345] TCP: Hash tables configured (established 65536 bind 65536) [ 0.739766] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.743165] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.746271] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.749310] NET: Registered protocol family 1 [ 0.752078] RPC: Registered named UNIX socket transport module. [ 0.754341] RPC: Registered udp transport module. [ 0.756249] RPC: Registered tcp transport module. [ 0.758240] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.760510] NET: Registered protocol family 44 [ 0.762252] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.764291] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.766118] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.768511] PCI: CLS 0 bytes, default 64 [ 0.770227] Unpacking initramfs... [ 2.080254] debug: unmapping init [mem 0xffff95c7fcc54000-0xffff95c7fffbffff] [ 2.084208] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.086286] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.088909] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.550990] Initialise system trusted keyrings [ 2.554412] Key type blacklist registered [ 2.556569] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.566583] zbud: loaded [ 2.569830] *** VALIDATE nfs *** [ 2.571305] *** VALIDATE nfs4 *** [ 2.573042] pstore: using deflate compression [ 2.576584] Platform Keyring initialized [ 2.681572] NET: Registered protocol family 38 [ 2.683436] Key type asymmetric registered [ 2.684948] Asymmetric key parser 'x509' registered [ 2.686906] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.690281] io scheduler mq-deadline registered [ 2.692078] io scheduler kyber registered [ 2.693761] io scheduler bfq registered [ 2.695651] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.698403] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.701197] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.703819] ACPI: Power Button [PWRF] [ 2.709285] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.716795] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.730824] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.738282] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.759740] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.786905] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.816773] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.822191] Non-volatile memory driver v1.3 [ 2.823352] Linux agpgart interface v0.103 [ 2.856738] virtio_blk virtio1: [vda] 149408 512-byte logical blocks (76.5 MB/73.0 MiB) [ 2.859872] vda: detected capacity change from 0 to 76496896 [ 2.882202] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.885177] vdb: detected capacity change from 0 to 1073741824 [ 2.906381] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.909701] vdc: detected capacity change from 0 to 2621440000 [ 2.944051] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.949095] vdd: detected capacity change from 0 to 2621440000 [ 2.979654] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.982033] vde: detected capacity change from 0 to 4294967296 [ 2.997202] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.000154] vdf: detected capacity change from 0 to 4294967296 [ 3.006980] libphy: Fixed MDIO Bus: probed [ 3.019244] usbcore: registered new interface driver usbserial_generic [ 3.022088] usbserial: USB Serial support registered for generic [ 3.024603] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.029675] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.031790] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.034212] mousedev: PS/2 mouse device common for all mice [ 3.037459] rtc_cmos 00:05: RTC can wake from S4 [ 3.040953] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.045117] rtc_cmos 00:05: registered as rtc0 [ 3.047559] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.048319] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.051443] intel_pstate: CPU model not supported [ 3.057071] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.061469] hid: raw HID events driver (C) Jiri Kosina [ 3.063843] usbcore: registered new interface driver usbhid [ 3.066176] usbhid: USB HID core driver [ 3.068118] drop_monitor: Initializing network drop monitor service [ 3.070304] Initializing XFRM netlink socket [ 3.073194] NET: Registered protocol family 10 [ 3.076342] Segment Routing with IPv6 [ 3.077765] NET: Registered protocol family 17 [ 3.079726] mpls_gso: MPLS GSO support [ 3.085995] RAS: Correctable Errors collector initialized. [ 3.088328] AVX version of gcm_enc/dec engaged. [ 3.090170] AES CTR mode by8 optimization enabled [ 3.168950] sched_clock: Marking stable (3168918063, 0)->(4072708090, -903790027) [ 3.172055] registered taskstats version 1 [ 3.173552] Loading compiled-in X.509 certificates [ 3.175233] zswap: loaded using pool lzo/zbud [ 3.201995] Key type big_key registered [ 3.216491] Key type encrypted registered [ 3.218204] ima: No TPM chip found, activating TPM-bypass! [ 3.220266] ima: Allocated hash algorithm: sha1 [ 3.222075] ima: No architecture policies found [ 3.223818] evm: Initialising EVM extended attributes: [ 3.225893] evm: security.selinux [ 3.227266] evm: security.ima [ 3.228579] evm: security.capability [ 3.229935] evm: HMAC attrs: 0x1 [ 3.232449] rtc_cmos 00:05: setting system clock to 2026-08-22 09:53:33 UTC (1787392413) [ 3.239131] debug: unmapping init [mem 0xffffffff91403000-0xffffffff915fffff] [ 3.242033] debug: unmapping init [mem 0xffffffff90182000-0xffffffff90458fff] [ 3.250102] Write protecting the kernel read-only data: 28672k [ 3.258040] debug: unmapping init [mem 0xffffffff8e803000-0xffffffff8e9fffff] [ 3.261483] debug: unmapping init [mem 0xffffffff8f114000-0xffffffff8f1fffff] [ 3.297827] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.305680] systemd[1]: Detected virtualization kvm. [ 3.307070] systemd[1]: Detected architecture x86-64. [ 3.308290] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.337151] systemd[1]: No hostname configured. [ 3.338459] systemd[1]: Set hostname to . [ 3.339879] random: systemd: uninitialized urandom read (16 bytes read) [ 3.341563] systemd[1]: Initializing machine ID from random generator. [ 3.393339] random: ln: uninitialized urandom read (6 bytes read) [ 3.491652] random: systemd: uninitialized urandom read (16 bytes read) [ 3.494902] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.498933] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.509273] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.265611] device-mapper: uevent: version 1.0.3 [ 4.269709] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.115644] random: fast init done [ 5.141470] virtio_net virtio0 ens2: renamed from eth0 [ 5.208903] scsi host0: ata_piix [ 5.258600] scsi host1: ata_piix [ 5.260420] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.263222] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.735812] random: crng init done [ 9.738816] random: 7 urandom warning(s) missed due to ratelimiting [ 9.772154] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.349440] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.521103] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.789214] SELinux: Disabled at runtime. [ 11.850947] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.860235] systemd[1]: Detected virtualization kvm. [ 11.862325] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.390487] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.394033] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.398682] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.403169] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.408516] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.415938] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.423341] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 12.458043] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.852860] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.145302] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.205752] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.322178] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.339940] EDAC sbridge: Ver: 1.1.2 [ 14.962836] Key type dns_resolver registered [ 15.250383] NFS: Registering the id_resolver key type [ 15.252155] Key type id_resolver registered [ 15.253705] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg218-server login: [ 40.142540] hrtimer: interrupt took 6530372 ns [ 70.657040] libcfs: loading out-of-tree module taints kernel. [ 70.706222] Key type ._llcrypt registered [ 70.708467] Key type .llcrypt registered [ 70.845653] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 88.822139] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 90.576952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 90.590754] alg: No test for adler32 (adler32-zlib) [ 92.103860] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 93.232119] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 95.159263] Key type lgssc registered [ 97.176437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.144531] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 158.750803] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 173.220138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 173.268473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.729560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 174.753644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 174.863631] Lustre: lustre-MDT0000: new disk, initializing [ 174.969197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.997876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 180.372982] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.196792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.330786] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.362694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 195.370766] Lustre: Skipped 1 previous similar message [ 195.514272] Lustre: lustre-MDT0001: new disk, initializing [ 195.614959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.644060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 195.653912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 200.256465] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.996643] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 214.321570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.538870] Lustre: lustre-OST0000: new disk, initializing [ 214.544047] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 214.550507] Lustre: 8453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.625934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.252481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.262070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.355186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 220.733111] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.114448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.318461] Lustre: lustre-OST0001: new disk, initializing [ 236.324944] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 236.333193] Lustre: 9526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.435966] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.836959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 244.852539] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.063255] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 245.283362] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.526442] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 268.455214] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 277.008575] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing check_logdir /tmp/testlogs/ [ 284.446866] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing yml_node [ 289.917283] Lustre: DEBUG MARKER: Client: 2.17.54.109 [ 293.253744] Lustre: DEBUG MARKER: MDS: 2.17.54.109 [ 296.672779] Lustre: DEBUG MARKER: OSS: 2.17.54.109 [ 298.610831] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 05:58:27 EDT 2026 [ 319.416137] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 321.058205] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.912536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 338.927915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.955794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.496753] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.502816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.512559] Lustre: Skipped 2 previous similar messages [ 342.518488] Lustre: Skipped 3 previous similar messages [ 347.620164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.638331] Lustre: Skipped 3 previous similar messages [ 352.736684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.754442] Lustre: Skipped 3 previous similar messages [ 353.247495] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 353.714822] Lustre: server umount lustre-MDT0000 complete [ 357.594057] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392767 with bad export cookie 16470665305404345031 [ 357.596456] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 357.609996] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 357.864317] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.890792] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 357.893253] Lustre: Skipped 1 previous similar message [ 357.913559] Lustre: Skipped 1 previous similar message [ 362.979797] LustreError: 10217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.992061] LustreError: 10217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 364.482066] Lustre: server umount lustre-MDT0001 complete [ 369.838242] Lustre: server umount lustre-OST0000 complete [ 374.831586] Lustre: server umount lustre-OST0001 complete [ 384.047605] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 397.285918] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 441.859505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.223758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 442.269741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 442.420773] Lustre: lustre-MDT0000: new disk, initializing [ 442.579349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.604485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 448.719619] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 463.119380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 463.249714] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 463.345145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 463.349507] Lustre: Skipped 1 previous similar message [ 463.444788] Lustre: lustre-MDT0001: new disk, initializing [ 463.572409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 463.632371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 463.655624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 468.808276] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 480.777456] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 487.011308] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 494.818117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 495.182464] Lustre: lustre-OST0000: new disk, initializing [ 495.189863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 495.206081] Lustre: 17067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 495.307035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.739664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 496.753642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 496.921565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 502.367610] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 512.600737] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 517.554455] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 517.765824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 522.890621] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 523.183879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 527.842615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.855820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 527.880255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 527.895036] Lustre: Skipped 3 previous similar messages [ 531.751573] Lustre: server umount lustre-OST0000 complete [ 538.083275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.108767] Lustre: Skipped 2 previous similar messages [ 548.323822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 548.330057] Lustre: Skipped 5 previous similar messages [ 550.367120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 550.558489] Lustre: server umount lustre-MDT0000 complete [ 553.445330] LustreError: 15094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.477559] LustreError: 15094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 555.575463] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392965 with bad export cookie 16470665305404346564 [ 555.580577] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.586357] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 555.962479] Lustre: server umount lustre-MDT0001 complete [ 562.754549] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 06:02:51 (1787392971) [ 585.116508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.686756] LustreError: 19943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.817229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 590.826157] LustreError: 19944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 591.947972] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 594.916594] LustreError: 19943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.376926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.122567] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 619.349496] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 626.503259] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 629.203384] Lustre: Failing over lustre-MDT0000 [ 629.454688] Lustre: server umount lustre-MDT0000 complete [ 629.729363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 629.750854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.766657] LustreError: 20672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.777195] LustreError: 20672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 649.184527] LustreError: 19939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 649.207244] LustreError: 19939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 650.207185] Lustre: 3648:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393044/real 1787393044] req@ffff95c743220000 x1874216881396608/t0(0) o400->MGC192.168.202.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787393060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 650.238174] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 650.414452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 660.448341] LustreError: 3646:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95c743df8380 x1874216881399040/t0(0) o250->MGC192.168.202.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 660.862586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 660.879402] Lustre: Skipped 1 previous similar message [ 660.954960] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 666.106073] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 666.118747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 666.173455] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 666.547903] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 678.564879] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 684.516098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.543725] Lustre: Skipped 1 previous similar message [ 684.553279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 684.557255] Lustre: Skipped 1 previous similar message [ 689.554483] Lustre: server umount lustre-MDT0000 complete [ 689.642156] LustreError: 19944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.672183] LustreError: 19944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 694.928686] LustreError: 19924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393105 with bad export cookie 16470665305404347978 [ 694.930909] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 694.948615] LustreError: 19924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 695.367599] Lustre: server umount lustre-MDT0001 complete [ 707.728154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.426282] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 728.224659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 734.986684] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 743.879198] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 749.689979] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 752.450612] Lustre: Failing over lustre-MDT0000 [ 752.461858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.475724] Lustre: Skipped 1 previous similar message [ 752.489989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 752.495859] Lustre: Skipped 1 previous similar message [ 752.678550] Lustre: server umount lustre-MDT0000 complete [ 758.758800] LustreError: 23287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 758.774321] LustreError: 23287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 768.992932] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393163/real 1787393163] req@ffff95c7426bce00 x1874216881427584/t0(0) o400->MGC192.168.202.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787393179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 769.027167] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 772.780682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.672652] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 785.900411] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 785.912783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 785.932371] Lustre: Skipped 1 previous similar message [ 785.968222] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 786.076610] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 797.487265] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 808.663456] Lustre: server umount lustre-MDT0000 complete [ 813.358266] LustreError: 23269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393223 with bad export cookie 16470665305404348916 [ 813.376611] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 813.881236] Lustre: server umount lustre-MDT0001 complete [ 825.208748] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 06:07:14 (1787393234) [ 848.132546] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 858.928683] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 904.417827] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 916.907433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.117969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 917.143335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 917.211570] Lustre: lustre-MDT0000: new disk, initializing [ 917.276908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.290500] Lustre: Skipped 3 previous similar messages [ 917.306994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 922.462301] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 934.551421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.708130] Lustre: 29673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 934.981230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 940.261293] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 952.040940] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 958.270344] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 966.067312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 966.287355] Lustre: 31644:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 968.263127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 968.333775] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 968.453277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 968.487330] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 973.709724] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 984.568755] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 991.083086] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1019.885363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1019.886193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.896214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1019.896226] Lustre: Skipped 2 previous similar messages [ 1019.953206] Lustre: Skipped 4 previous similar messages [ 1030.113749] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1030.534216] Lustre: server umount lustre-OST0000 complete [ 1037.280527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1041.874693] Lustre: server umount lustre-MDT0000 complete [ 1045.476035] LustreError: 31236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.496162] LustreError: 31236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 1047.085853] LustreError: 29666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393457 with bad export cookie 16470665305404349322 [ 1047.102263] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1047.112218] LustreError: 29666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1047.786780] Lustre: server umount lustre-MDT0001 complete [ 1061.151514] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1064.732384] Key type lgssc unregistered [ 1065.021428] LNet: 33625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.039897] LNetError: 33625:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.067072] LNet: Removed LNI 192.168.202.118@tcp [ 1066.413203] Key type .llcrypt unregistered [ 1066.417248] Key type ._llcrypt unregistered [ 1096.588828] Key type ._llcrypt registered [ 1096.592643] Key type .llcrypt registered [ 1096.727639] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1098.679662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1098.723055] alg: No test for adler32 (adler32-zlib) [ 1100.243572] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 1101.034123] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1102.953819] Key type lgssc registered [ 1105.358051] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1115.587175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1115.627259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.157113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1122.173447] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1127.395851] LustreError: 35045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.448947] LustreError: 35045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1132.526241] LustreError: 35044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1132.574311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.994014] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1137.971832] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1149.270271] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1156.235981] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1164.984800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1165.525854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.609385] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1177.843527] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1181.480898] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1187.108193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1202.586658] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 06:13:31 (1787393611) [ 1204.691359] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1206.804912] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1215.848651] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 06:13:44 (1787393624) [ 1226.280611] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 06:13:55 (1787393635) [ 1227.863362] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1239.023332] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1239.032140] Lustre: Skipped 2 previous similar messages [ 1241.460613] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 06:14:09 (1787393649) [ 1243.658905] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1243.671814] Lustre: Skipped 2 previous similar messages [ 1252.750730] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1252.762239] Lustre: Skipped 4 previous similar messages [ 1269.845503] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 1269.856537] Lustre: Skipped 5 previous similar messages [ 1272.321049] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 06:14:40 (1787393680) [ 1276.000189] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1280.503088] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1290.999634] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 06:14:59 (1787393699) [ 1305.902711] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1305.908189] Lustre: Skipped 4 previous similar messages [ 1307.849600] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 06:15:16 (1787393716) [ 1325.547496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1325.547530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.555872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.566369] Lustre: Skipped 1 previous similar message [ 1325.657270] Lustre: Skipped 1 previous similar message [ 1328.708958] Lustre: server umount lustre-OST0000 complete [ 1335.777906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.801547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1339.226995] Lustre: server umount lustre-MDT0000 complete [ 1340.896721] LustreError: 35044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.911358] LustreError: 35044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1343.368140] LustreError: 35023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393753 with bad export cookie 2580093748936141955 [ 1343.374958] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1343.383650] LustreError: 35023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1343.921086] Lustre: server umount lustre-MDT0001 complete [ 1356.194442] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1359.083853] Key type lgssc unregistered [ 1359.500531] LNet: 42913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.511448] LNetError: 42913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.526798] LNet: Removed LNI 192.168.202.118@tcp [ 1360.447766] Key type .llcrypt unregistered [ 1360.450370] Key type ._llcrypt unregistered [ 1381.016959] Key type ._llcrypt registered [ 1381.019852] Key type .llcrypt registered [ 1381.122621] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 1400.964101] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1402.099962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1402.121430] alg: No test for adler32 (adler32-zlib) [ 1403.413538] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 1403.885612] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1405.663179] Key type lgssc registered [ 1406.748933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1446.726432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1446.767142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.134405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1448.224774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1448.344750] Lustre: lustre-MDT0000: new disk, initializing [ 1448.480306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.516719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1453.182790] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1464.334856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.471582] Lustre: 46083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1464.522687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1464.526719] Lustre: Skipped 1 previous similar message [ 1464.600894] Lustre: lustre-MDT0001: new disk, initializing [ 1464.706982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1464.739174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1464.745985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1469.529445] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1479.715811] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1486.427169] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1493.405933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1493.708649] Lustre: lustre-OST0000: new disk, initializing [ 1493.712778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1493.717809] Lustre: 48051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1493.846951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1501.172477] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1503.840076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1503.854557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1504.003666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1511.981572] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1517.426862] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1517.615750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1522.972735] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1523.173519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1529.824897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1529.826590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1529.840824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1529.872189] Lustre: Skipped 1 previous similar message [ 1531.260164] Lustre: server umount lustre-OST0000 complete [ 1536.482733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1536.496839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1536.504420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1536.508186] Lustre: Skipped 1 previous similar message [ 1540.064671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1540.066201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.770427] Lustre: server umount lustre-MDT0000 complete [ 1545.804901] LustreError: 46076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393956 with bad export cookie 2218953013808426486 [ 1545.809954] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1545.812481] LustreError: 46076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1546.258228] Lustre: server umount lustre-MDT0001 complete [ 1557.088790] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 06:19:25 (1787393965) [ 1564.244793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.582229] LustreError: 50512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.596669] LustreError: 50512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1564.646269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.550610] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1569.761214] LustreError: 50513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1573.860969] LustreError: 50512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1578.714791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.001576] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1579.022016] Lustre: Skipped 1 previous similar message [ 1579.142381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1584.437361] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1592.756772] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1599.214817] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1606.271092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1606.816347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1613.298791] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1622.209415] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1630.219875] LustreError: 53041:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1631.543605] Lustre: Modifying parameter general.timeout=1 in log params [ 1632.603974] Lustre: Modifying parameter general.timeout=2 in log params [ 1634.891245] Lustre: Modifying parameter general.timeout=4 in log params [ 1634.902488] Lustre: Skipped 1 previous similar message [ 1639.250980] Lustre: Modifying parameter general.timeout=8 in log params [ 1639.254315] Lustre: Skipped 3 previous similar messages [ 1647.898905] Lustre: Modifying parameter general.timeout=16 in log params [ 1647.902192] Lustre: Skipped 7 previous similar messages [ 1663.937478] Lustre: Modifying parameter general.timeout=35 in log params [ 1663.940887] Lustre: Skipped 18 previous similar messages [ 1696.423130] Lustre: Modifying parameter general.timeout=69 in log params [ 1696.431707] Lustre: Skipped 33 previous similar messages [ 1763.908728] Lustre: Disabling parameter general.timeout= in log params [ 1763.913708] Lustre: Skipped 60 previous similar messages [ 1767.904991] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 06:22:56 (1787394176) [ 1771.230700] Lustre: 59989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1772.995103] Lustre: 60098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1775.177637] Lustre: 60196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1779.330839] Lustre: 60393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1779.345424] Lustre: 60393:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1813.599888] Lustre: 61864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1813.607423] Lustre: 61864:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1813.653472] LustreError: 61864:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1819.803219] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 06:23:48 (1787394228) [ 1834.980323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1834.994125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1835.005025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1835.012504] Lustre: Skipped 2 previous similar messages [ 1835.031185] Lustre: Skipped 2 previous similar messages [ 1839.527626] Lustre: server umount lustre-MDT0000 complete [ 1840.103324] LustreError: 50508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1840.127529] LustreError: 50508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1843.772926] LustreError: 50493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394254 with bad export cookie 2218953013808427368 [ 1843.778441] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1843.783631] LustreError: 50493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1844.041723] Lustre: server umount lustre-MDT0001 complete [ 1858.942326] Lustre: server umount lustre-OST0000 complete [ 1894.289228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.649930] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1894.674493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1894.680757] Lustre: Skipped 22 previous similar messages [ 1894.897685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.822859] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1911.019846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.124774] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1911.157460] Lustre: 64398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1918.083566] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1928.282559] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1936.007567] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1944.634056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1944.860780] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1945.266732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.269719] Lustre: Skipped 1 previous similar message [ 1953.547957] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1962.182074] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1964.156164] Lustre: 66968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1972.194293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1972.232611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.273011] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.285094] Lustre: Skipped 1 previous similar message [ 1977.325675] LustreError: 66248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.361327] LustreError: 66248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1977.434534] Lustre: server umount lustre-OST0000 complete [ 1982.443327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.457306] Lustre: Skipped 2 previous similar messages [ 1992.672758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1992.692602] Lustre: Skipped 5 previous similar messages [ 1996.255163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1996.489326] Lustre: server umount lustre-MDT0000 complete [ 1997.793927] LustreError: 64410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1997.821934] LustreError: 64410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2002.640453] LustreError: 65140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394412 with bad export cookie 2218953013808430511 [ 2002.658113] LustreError: 65140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2002.659847] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2003.279200] Lustre: server umount lustre-MDT0001 complete [ 2014.514614] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2017.672799] Key type lgssc unregistered [ 2018.016988] LNet: 68179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2018.024441] LNetError: 68179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2018.079064] LNet: Removed LNI 192.168.202.118@tcp [ 2019.144983] Key type .llcrypt unregistered [ 2019.149351] Key type ._llcrypt unregistered [ 2043.752449] Key type ._llcrypt registered [ 2043.755440] Key type .llcrypt registered [ 2043.844478] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2044.878609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2045.060733] alg: No test for adler32 (adler32-zlib) [ 2046.059417] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 2046.279454] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2047.951186] Key type lgssc registered [ 2049.284447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2057.123954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2057.148273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.616886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.675982] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2067.937095] LustreError: 69593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.954749] LustreError: 69593:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2072.892294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.056584] LustreError: 69594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.594535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2078.869615] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2087.451695] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2093.997807] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2101.588142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2102.111960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.926861] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2118.149055] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2127.174972] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2139.116126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.125474] Lustre: Skipped 1 previous similar message [ 2139.126478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.135121] Lustre: Skipped 2 previous similar messages [ 2142.593211] Lustre: server umount lustre-MDT0000 complete [ 2144.231662] LustreError: 69590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.257920] LustreError: 69590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2145.998104] LustreError: 70335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394556 with bad export cookie 13440371647842284828 [ 2146.007177] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2146.022086] LustreError: 70335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2146.591675] Lustre: server umount lustre-MDT0001 complete [ 2162.454943] Lustre: server umount lustre-OST0000 complete [ 2195.522537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.768253] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2195.798610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2196.035210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.939997] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2211.253544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.394771] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2211.467451] Lustre: 74464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2212.177992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2217.377377] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2225.945500] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2232.317844] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2241.345620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2241.577504] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2242.285403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.462525] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2259.174884] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2261.179272] Lustre: 77041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2263.158537] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2272.223988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2272.238061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.253908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2272.738134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2275.213817] Lustre: server umount lustre-OST0000 complete [ 2282.979461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2282.989050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2283.003927] Lustre: Skipped 2 previous similar messages [ 2283.030108] Lustre: Skipped 2 previous similar messages [ 2285.768146] Lustre: server umount lustre-MDT0000 complete [ 2288.101996] LustreError: 76762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.122837] LustreError: 76762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2289.768825] LustreError: 74456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394700 with bad export cookie 13440371647842285675 [ 2289.774392] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2289.782975] LustreError: 74456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2290.319785] Lustre: server umount lustre-MDT0001 complete [ 2306.269879] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2310.718558] Key type lgssc unregistered [ 2311.084692] LNet: 78252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.103157] LNetError: 78252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2311.121755] LNet: Removed LNI 192.168.202.118@tcp [ 2312.871639] Key type .llcrypt unregistered [ 2312.874463] Key type ._llcrypt unregistered [ 2343.123343] Key type ._llcrypt registered [ 2343.126492] Key type .llcrypt registered [ 2343.256227] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2344.385326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2344.414887] alg: No test for adler32 (adler32-zlib) [ 2345.638880] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 2345.843187] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2347.536283] Key type lgssc registered [ 2348.719546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2356.967396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2356.989327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.805141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.960955] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2367.979284] LustreError: 79667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.013231] LustreError: 79667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2373.091597] LustreError: 79668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.547433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.494147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2381.967472] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2391.120307] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2398.101817] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2405.902351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2406.566298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.134356] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2419.929949] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2435.380116] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2436.286176] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2437.921960] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2437.927822] Lustre: Skipped 1 previous similar message [ 2440.456872] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2440.465036] Lustre: Skipped 2 previous similar messages [ 2445.306377] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2445.314891] Lustre: Skipped 4 previous similar messages [ 2453.971264] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2453.981481] Lustre: Skipped 9 previous similar messages [ 2470.161884] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2470.165347] Lustre: Skipped 16 previous similar messages [ 2502.487743] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2502.493744] Lustre: Skipped 32 previous similar messages [ 2542.296064] Lustre: 87797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2546.006540] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 06:35:54 (1787394954) [ 2547.970741] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2549.794785] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 06:35:59 (1787394959) [ 2556.392865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.399662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2556.409968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2556.421047] Lustre: Skipped 1 previous similar message [ 2560.916732] Lustre: server umount lustre-OST0000 complete [ 2565.610061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2565.620796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.646805] Lustre: Skipped 1 previous similar message [ 2565.654991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2566.628691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.666614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2566.673674] Lustre: Skipped 1 previous similar message [ 2571.644888] Lustre: server umount lustre-MDT0000 complete [ 2571.747264] LustreError: 79664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2571.760657] LustreError: 79664:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2575.656335] LustreError: 82281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394985 with bad export cookie 6602394362126503168 [ 2575.672298] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.085350] Lustre: server umount lustre-MDT0001 complete [ 2592.020045] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2595.401375] Key type lgssc unregistered [ 2595.670736] LNet: 89048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2595.682298] LNetError: 89048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2595.703724] LNet: Removed LNI 192.168.202.118@tcp [ 2596.771722] Key type .llcrypt unregistered [ 2596.773551] Key type ._llcrypt unregistered [ 2602.320473] Key type ._llcrypt registered [ 2602.327303] Key type .llcrypt registered [ 2602.527187] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2608.002915] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules [ 2609.333386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2609.379718] alg: No test for adler32 (adler32-zlib) [ 2610.452565] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 2610.752484] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2612.479187] Key type lgssc registered [ 2613.843816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2615.136200] LustreError: 90256:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2633.427509] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2636.708703] LustreError: 90256:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2644.343147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2644.372171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2646.036043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2652.385985] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2656.253150] LustreError: 91250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.284133] LustreError: 91250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2661.346468] LustreError: 91251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2664.990122] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 06:37:53 (1787395073) [ 2666.464642] LustreError: 91250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.861206] LustreError: 91282:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2670.883251] LustreError: 91282:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2670.897760] LustreError: 91282:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 25, retries 0, failed: rc = -5 [ 2671.666863] Lustre: server umount lustre-MDT0000 complete [ 2686.492854] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2688.986271] Key type lgssc unregistered [ 2689.363904] LNet: 92777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2689.370384] LNetError: 92777:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2690.406385] LNet: Removed LNI 192.168.202.118@tcp [ 2691.244312] Key type .llcrypt unregistered [ 2691.246255] Key type ._llcrypt unregistered [ 2714.439844] Key type ._llcrypt registered [ 2714.447979] Key type .llcrypt registered [ 2714.615842] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2715.586953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2715.623168] alg: No test for adler32 (adler32-zlib) [ 2716.999524] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 2717.272375] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2719.047175] Key type lgssc registered [ 2720.328381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2728.386135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2728.423470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.163986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.722343] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2740.707664] LustreError: 93996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2740.729938] LustreError: 93996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2744.079420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.795070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2749.301824] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2758.086045] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2765.866836] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2773.967341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2774.887750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2785.074335] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2795.836960] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2852.130483] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 06:41:00 (1787395260) [ 2854.159411] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2856.513730] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 06:41:05 (1787395265) [ 2866.146100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.163946] Lustre: Skipped 2 previous similar messages [ 2866.165339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2866.181113] Lustre: Skipped 2 previous similar messages [ 2868.921641] Lustre: server umount lustre-MDT0000 complete [ 2871.274660] LustreError: 93993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.304900] LustreError: 93993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2873.816106] LustreError: 93976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395284 with bad export cookie 51118863792573329 [ 2873.836714] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2873.840740] LustreError: 93976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2874.336814] Lustre: server umount lustre-MDT0001 complete [ 2889.971568] Lustre: server umount lustre-OST0000 complete [ 2906.077257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.802862] LustreError: 98294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2907.087622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.220362] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2912.249157] LustreError: 98295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2912.272820] LustreError: 98295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2917.347218] LustreError: 98294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2924.645942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.452679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2930.211620] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2939.683340] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2946.275647] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2962.323771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2962.619067] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2962.623990] LustreError: 98284:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2962.646230] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2962.658476] LustreError: 100379:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2962.669381] LustreError: 100379:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2962.681347] LustreError: 100379:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2962.757480] Lustre: server umount lustre-OST0000 complete [ 2962.761322] LustreError: 100379:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2972.510831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2972.715875] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2972.726219] LustreError: 98284:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2972.748077] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2972.754563] LustreError: 100846:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2972.760889] LustreError: 100846:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2972.768452] LustreError: 100846:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2972.830150] Lustre: server umount lustre-OST0000 complete [ 2972.832541] LustreError: 100846:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2982.863613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2983.018069] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2983.021371] Lustre: Found index 0 for lustre-OST0000, updating log [ 2983.025504] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2983.105980] Lustre: lustre-OST0000: new disk, initializing [ 2983.108638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2983.115943] Lustre: 101381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2983.503796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2991.663975] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2995.934038] Lustre: Failing over lustre-OST0000 [ 2996.100918] Lustre: server umount lustre-OST0000 complete [ 3000.385474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.388520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3000.415128] Lustre: Skipped 1 previous similar message [ 3005.409977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.412179] Lustre: Skipped 2 previous similar messages [ 3006.851331] Lustre: server umount lustre-MDT0000 complete [ 3010.528273] LustreError: 98295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3010.562809] LustreError: 98295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3011.020363] LustreError: 98277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395421 with bad export cookie 51118863792576073 [ 3011.026185] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3011.037428] LustreError: 98277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3021.470444] Lustre: server umount lustre-MDT0001 complete [ 3032.508629] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 06:44:00 (1787395440) [ 3052.875676] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3063.804671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3064.289158] LustreError: 103730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3064.436461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3069.489839] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3079.365531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.967049] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3089.971402] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3097.070431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3097.709277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.712373] Lustre: Skipped 1 previous similar message [ 3097.732107] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3104.758366] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3104.881992] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3104.925665] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3104.926593] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3113.823960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3113.906726] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3113.962671] Lustre: lustre-OST0001: new disk, initializing [ 3113.966538] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3113.971677] Lustre: 106046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3120.561995] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3123.815261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3123.832960] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3123.943586] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3127.880990] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3134.091891] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3139.561405] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.574380] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3139.583556] Lustre: Skipped 2 previous similar messages [ 3139.607785] Lustre: Skipped 2 previous similar messages [ 3143.621816] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [ 3145.016979] Lustre: server umount lustre-MDT0001 complete [ 3148.725378] LustreError: 103727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.745480] LustreError: 103727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3156.019335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.109906] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3156.117491] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3156.157984] Lustre: 103719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3157.935514] LustreError: 107591:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3157.945625] LustreError: 107591:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3157.958860] LustreError: 103722:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3158.107158] LustreError: 107602:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.118@tcp: cfg command failed: rc = -17 [ 3158.118821] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3158.118821] [ 3161.946571] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3162.606375] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3172.755790] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 06:46:21 (1787395581) [ 3192.742863] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3211.652727] Lustre: 109667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3235.427321] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3239.164969] Lustre: 110803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3330.533398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3330.541895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3330.552165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3330.559119] Lustre: Skipped 3 previous similar messages [ 3336.223353] Lustre: server umount lustre-MDT0000 complete [ 3336.676936] LustreError: 108345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.695016] LustreError: 108345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 3339.966483] LustreError: 106768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395750 with bad export cookie 51118863792577074 [ 3339.976544] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3339.984639] LustreError: 106768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3340.414620] Lustre: server umount lustre-MDT0001 complete [ 3355.398536] Lustre: server umount lustre-OST0000 complete [ 3358.307382] Lustre: 93306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395752/real 1787395752] req@ffff95c74595b800 x1874219633068032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787395768 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3358.332498] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3358.342524] Lustre: Skipped 3 previous similar messages [ 3359.230915] Lustre: server umount lustre-OST0001 complete [ 3365.218864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3375.889091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3384.860482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3396.772960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3418.423867] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3431.223486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3431.252899] Lustre: lustre-MDT0000: reset Object Index mappings [ 3432.016325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3432.032151] Lustre: Skipped 2 previous similar messages [ 3437.520288] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3447.344645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3447.367592] Lustre: lustre-MDT0001: reset Object Index mappings [ 3453.269614] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3456.317637] Lustre: 115901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3463.975739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3471.630768] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3472.809429] LustreError: 116255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3472.825462] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3472.828461] LustreError: 116255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3476.926512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3481.913098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3487.749210] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3487.770656] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3489.741650] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3499.458874] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3504.109170] Lustre: 117780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3562.371646] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 06:52:51 (1787395971) [ 3590.114833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3590.137262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3590.143495] Lustre: Skipped 7 previous similar messages [ 3594.725329] Lustre: server umount lustre-MDT0000 complete [ 3600.090390] LustreError: 117800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396010 with bad export cookie 51118863792623967 [ 3600.109195] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3600.116700] LustreError: 117800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3606.985391] Lustre: server umount lustre-MDT0001 complete [ 3619.174262] Lustre: server umount lustre-OST0000 complete [ 3630.464627] Lustre: server umount lustre-OST0001 complete [ 3639.476828] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 3653.650282] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3698.468181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3698.784560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3698.821759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3698.928931] Lustre: lustre-MDT0000: new disk, initializing [ 3699.016468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3699.020350] Lustre: Skipped 3 previous similar messages [ 3699.034200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3703.388817] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3710.644878] Lustre: server umount lustre-MDT0000 complete [ 3723.830749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.218842] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3724.233175] Lustre: 122051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3724.253254] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3729.559412] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3735.250704] Lustre: server umount lustre-MDT0000 complete [ 3745.613835] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 06:55:54 (1787396154) [ 3747.668228] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3749.626366] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 06:55:58 (1787396158) [ 3771.959935] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3783.726854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3789.660097] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3799.765472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.895902] Lustre: 124125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3799.951431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3799.956103] Lustre: Skipped 1 previous similar message [ 3800.047762] Lustre: lustre-MDT0001: new disk, initializing [ 3800.185768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3800.210039] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3804.820422] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3810.440804] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3818.358075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3818.787787] Lustre: lustre-OST0000: new disk, initializing [ 3818.790875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3818.796381] Lustre: 125701:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3820.242356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3820.257853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3820.360244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3825.741470] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3837.522708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3837.683514] Lustre: lustre-OST0001: new disk, initializing [ 3837.686431] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3837.692462] Lustre: 126571:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3838.921694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3838.929718] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3839.037625] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3844.532266] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3855.240415] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3859.945739] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4289.485597] Lustre: 133001:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4289.493880] Lustre: 133001:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4289.502915] Lustre: 133001:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4289.534383] Lustre: 133001:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4289.538513] Lustre: 133001:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4289.543438] Lustre: 133001:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4625.450637] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:10:33 (1787397033) [ 4642.788077] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4642.792030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4642.827708] Lustre: Skipped 6 previous similar messages [ 4642.838047] Lustre: Skipped 4 previous similar messages [ 4643.691214] Lustre: server umount lustre-MDT0000 complete [ 4647.905687] LustreError: 124946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4647.926400] LustreError: 124946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 4648.597817] LustreError: 124863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397058 with bad export cookie 51118863792693379 [ 4648.606038] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4648.617672] LustreError: 124863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4653.026647] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4653.037079] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4653.042879] Lustre: Skipped 3 previous similar messages [ 4653.057214] Lustre: Skipped 4 previous similar messages [ 4655.677473] Lustre: server umount lustre-MDT0001 complete [ 4666.890564] Lustre: server umount lustre-OST0000 complete [ 4677.169815] Lustre: server umount lustre-OST0001 complete [ 4683.352754] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 4693.470943] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4737.213762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4737.405925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4737.437916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4737.534903] Lustre: lustre-MDT0000: new disk, initializing [ 4737.655544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4737.664801] Lustre: Skipped 5 previous similar messages [ 4737.699916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4742.673403] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4755.538075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.788487] Lustre: 136064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4755.857425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4755.867595] Lustre: Skipped 1 previous similar message [ 4756.015740] Lustre: lustre-MDT0001: new disk, initializing [ 4756.253368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4756.274410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4762.228646] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4774.606986] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4782.174676] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4790.530741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4790.874967] Lustre: lustre-OST0000: new disk, initializing [ 4790.878349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4790.888346] Lustre: 138033:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4791.029273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4791.034730] Lustre: Skipped 1 previous similar message [ 4792.528649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4792.548844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4792.673348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4797.837607] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4807.532083] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4819.751620] Lustre: lustre-MDD0000: changelog on [ 4822.529373] Lustre: lustre-MDD0001: changelog on [ 6859.302804] Lustre: lustre-MDD0001: changelog off [ 6860.984378] Lustre: 136070:0:(llog_cat.c:984:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6862.170229] Lustre: lustre-MDD0000: changelog off [ 6869.485392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6869.523718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6869.535770] Lustre: Skipped 1 previous similar message [ 6874.596031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6874.605615] Lustre: Skipped 1 previous similar message [ 6879.734240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6879.742806] Lustre: Skipped 1 previous similar message [ 6883.295225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6883.720141] Lustre: server umount lustre-OST0000 complete [ 6889.962384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6889.971000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6889.991532] Lustre: Skipped 2 previous similar messages [ 6890.015093] Lustre: Skipped 1 previous similar message [ 6895.072058] LustreError: 136074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6895.102580] LustreError: 136074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6895.156801] Lustre: server umount lustre-MDT0000 complete [ 6899.425044] LustreError: 136055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399309 with bad export cookie 51118863793519575 [ 6899.446077] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6899.952462] Lustre: server umount lustre-MDT0001 complete [ 6915.482900] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 6919.002837] Key type lgssc unregistered [ 6919.286480] LNet: 147202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6919.298162] LNetError: 147202:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6919.327291] LNet: Removed LNI 192.168.202.118@tcp [ 6920.280154] Key type .llcrypt unregistered [ 6920.281877] Key type ._llcrypt unregistered [ 6941.107587] Key type ._llcrypt registered [ 6941.113528] Key type .llcrypt registered [ 6941.203783] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 6958.841566] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 6960.095143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6960.219389] alg: No test for adler32 (adler32-zlib) [ 6961.310545] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 6961.501483] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 6963.247618] Key type lgssc registered [ 6964.314374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6998.835612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6998.894370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7000.096871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7000.118602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7000.175378] Lustre: lustre-MDT0000: new disk, initializing [ 7000.224844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7000.243588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7004.507607] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7014.996284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7015.116965] Lustre: 150405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7015.168758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7015.172836] Lustre: Skipped 1 previous similar message [ 7015.237530] Lustre: lustre-MDT0001: new disk, initializing [ 7015.325572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7015.359346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7015.380368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7019.252799] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7029.199826] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7034.722594] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7041.256960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7041.509218] Lustre: lustre-OST0000: new disk, initializing [ 7041.512386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7041.518937] Lustre: 152371:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7041.604161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7047.732374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7047.738337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7047.862162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7048.155109] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7057.123308] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7061.951532] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_module obdecho/obdecho [ 7063.635493] LustreError: 153375:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7063.650583] LustreError: 153375:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7065.274780] LustreError: 153471:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 7078.884884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7078.890047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7078.903154] Lustre: Skipped 1 previous similar message [ 7083.399409] Lustre: server umount lustre-MDT0000 complete [ 7086.711577] LustreError: 150397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399496 with bad export cookie 15458004885076905516 [ 7086.714634] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7086.720546] LustreError: 150397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7087.017149] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7087.037223] Lustre: Skipped 1 previous similar message [ 7087.114354] Lustre: server umount lustre-MDT0001 complete [ 7093.180994] Lustre: server umount lustre-OST0000 complete [ 7114.623847] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 7126.617475] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7165.022714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7165.329968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7165.415645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7165.532348] Lustre: lustre-MDT0000: new disk, initializing [ 7165.624940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7165.647910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7170.651122] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7184.514653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7184.795289] Lustre: 157186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7184.912078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7184.925046] Lustre: Skipped 1 previous similar message [ 7185.100860] Lustre: lustre-MDT0001: new disk, initializing [ 7185.224616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7185.293208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7185.326623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7192.969883] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7205.293768] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7212.293283] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7218.897567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7219.160472] Lustre: lustre-OST0000: new disk, initializing [ 7219.163932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7219.169716] Lustre: 159155:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7219.238464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7225.180095] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7226.445395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7226.458201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7226.553467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7234.399932] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7240.086385] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7240.380262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7245.988697] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7246.225181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7252.456933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.464285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7252.486509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7252.501228] Lustre: Skipped 3 previous similar messages [ 7254.353522] Lustre: server umount lustre-OST0000 complete [ 7257.579928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7257.591377] Lustre: Skipped 1 previous similar message [ 7257.599457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7262.178242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7262.192705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.218183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7263.795878] Lustre: server umount lustre-MDT0000 complete [ 7267.809864] LustreError: 157197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7267.821408] LustreError: 157177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399678 with bad export cookie 15458004885076906482 [ 7267.838527] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7267.841949] LustreError: 157197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 7267.884304] LustreError: 157177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7268.134574] Lustre: server umount lustre-MDT0001 complete [ 7271.949268] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 07:54:40 (1787399680) [ 7273.589218] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7275.736510] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 07:54:44 (1787399684) [ 7282.395551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.756278] LustreError: 161568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7282.825980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7286.721515] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7288.290176] LustreError: 161569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7293.415290] LustreError: 161568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7294.379479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7298.650074] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7306.799609] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7313.762903] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7320.740171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7321.107966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7321.113832] Lustre: Skipped 1 previous similar message [ 7327.617782] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7335.705884] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7347.125764] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 7347.131377] Lustre: Skipped 1 previous similar message [ 7348.533404] Lustre: server umount lustre-MDT0000 complete [ 7349.734079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7349.744569] Lustre: Skipped 1 previous similar message [ 7349.748196] LustreError: 161568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7352.364076] LustreError: 161548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399762 with bad export cookie 15458004885076907364 [ 7352.375044] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7352.385945] LustreError: 161548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7357.378649] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [ 7357.383824] Lustre: Skipped 1 previous similar message [ 7357.861665] LustreError: 161564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7357.890951] LustreError: 161564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7359.088330] Lustre: server umount lustre-MDT0001 complete [ 7365.285231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7377.305488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7377.696239] LustreError: 165413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7377.698544] LustreError: 165408:0:(llog.c:1655:llog_backup()) MGC192.168.202.118@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7377.712675] Lustre: 165408:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.118@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7412.192501] LustreError: 165414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7412.224297] LustreError: 165414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 7438.304487] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7438.311227] Lustre: 165408:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.118@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7477.731031] LustreError: 165433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7477.747893] LustreError: 165433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 7479.295473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7483.836962] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7484.393182] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7492.106749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.705330] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd685dd8aff432e9a [ 7497.716198] Lustre: MGC192.168.202.118@tcp: Connection restored to 0@lo (at 0@lo) [ 7501.622606] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7503.342476] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7509.073069] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7510.794353] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7516.825670] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7518.453169] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7539.169289] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.195343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7539.195533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7539.195537] Lustre: Skipped 1 previous similar message [ 7539.197676] Lustre: Skipped 3 previous similar messages [ 7540.900846] Lustre: server umount lustre-MDT0000 complete [ 7544.481708] LustreError: 161548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399954 with bad export cookie 15458004885076913818 [ 7544.485665] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7544.495772] LustreError: 161548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7544.818569] Lustre: server umount lustre-MDT0001 complete [ 7557.944460] Lustre: server umount lustre-OST0000 complete [ 7580.744587] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 7590.909384] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7627.967827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.175162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7628.195986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7628.247934] Lustre: lustre-MDT0000: new disk, initializing [ 7628.329598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7628.333722] Lustre: Skipped 1 previous similar message [ 7628.349726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7633.896808] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7647.280201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7647.372866] Lustre: 170510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7647.660637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7652.559601] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7663.799125] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7670.943200] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7678.042512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7678.526068] Lustre: lustre-OST0000: new disk, initializing [ 7678.529945] Lustre: Skipped 1 previous similar message [ 7678.536293] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7678.549930] Lustre: Skipped 2 previous similar messages [ 7678.561393] Lustre: 172493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7685.113640] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7688.289467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7688.300780] Lustre: Skipped 1 previous similar message [ 7688.308100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7688.433023] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7694.990824] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7700.183796] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7700.427878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7705.116951] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7705.337258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7709.153976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7709.154415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7709.173114] LustreError: Skipped 1 previous similar message [ 7709.173756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7709.173760] Lustre: Skipped 2 previous similar messages [ 7709.225172] Lustre: Skipped 2 previous similar messages [ 7713.655169] Lustre: server umount lustre-OST0000 complete [ 7719.400386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7723.908266] Lustre: server umount lustre-MDT0000 complete [ 7724.513094] LustreError: 170523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7724.533235] LustreError: 170523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 7728.083979] LustreError: 170503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400138 with bad export cookie 15458004885076930653 [ 7728.086926] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7728.092504] LustreError: 170503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7728.703895] Lustre: server umount lustre-MDT0001 complete [ 7737.520929] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:02:26 (1787400146) [ 7744.536334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7749.893639] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7759.591840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7765.126680] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7772.671457] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7778.584584] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7786.006918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7794.032517] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7804.732803] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7815.980468] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:03:44 (1787400224) [ 7820.261224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7820.266619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7820.270390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7820.270398] Lustre: Skipped 3 previous similar messages [ 7820.275638] Lustre: Skipped 3 previous similar messages [ 7826.368484] Lustre: server umount lustre-OST0000 complete [ 7844.319227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7844.551831] Lustre: server umount lustre-MDT0000 complete [ 7849.068615] LustreError: 174883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400259 with bad export cookie 15458004885076931535 [ 7849.075796] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7849.085802] LustreError: 174883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7849.483660] Lustre: server umount lustre-MDT0001 complete [ 7863.398260] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 7866.489720] Key type lgssc unregistered [ 7866.937289] LNet: 178634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7866.947481] LNetError: 178634:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7866.973599] LNet: Removed LNI 192.168.202.118@tcp [ 7868.027944] Key type .llcrypt unregistered [ 7868.031927] Key type ._llcrypt unregistered [ 7891.007222] Key type ._llcrypt registered [ 7891.009073] Key type .llcrypt registered [ 7891.161716] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7892.480599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7892.514956] alg: No test for adler32 (adler32-zlib) [ 7893.673310] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 7894.006786] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 7895.743164] Key type lgssc registered [ 7896.961220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7904.547747] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7904.560347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7920.095411] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7925.215319] LustreError: 179884:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.118@tcp: failed processing log, type 4: rc = -110 [ 7930.337328] LustreError: 179884:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7930.348666] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7930.352865] LustreError: MGC192.168.202.118@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7930.367476] LustreError: 179884:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7930.376830] LustreError: 179884:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7930.382142] LustreError: 179884:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7930.439185] Lustre: server umount lustre-OST0000 complete [ 7930.442774] LustreError: 179884:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7938.478219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7939.967687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7944.489410] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7950.305599] LustreError: 180382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7950.340988] LustreError: 180382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7953.478314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7953.803348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7958.027467] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7966.197526] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7971.544152] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7978.710490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7978.898666] LustreError: 182201:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7978.909827] Lustre: 182201:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.118@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7979.069075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7987.176590] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7995.486347] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7999.968873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7999.972217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7999.986207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8000.997590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8000.998583] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8003.887522] Lustre: server umount lustre-MDT0000 complete [ 8006.114040] LustreError: 181403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8006.149629] LustreError: 181403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8007.901897] LustreError: 180363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400418 with bad export cookie 8071444625943475491 [ 8007.909312] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8007.911812] LustreError: 180363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8008.396524] Lustre: server umount lustre-MDT0001 complete [ 8022.018467] Lustre: server umount lustre-OST0000 complete [ 8032.236126] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 8042.902401] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8079.792248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8080.033287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8080.059907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8080.142686] Lustre: lustre-MDT0000: new disk, initializing [ 8080.238572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8080.252651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8084.900783] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8095.197442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8095.281200] Lustre: 185724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8095.318554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8095.321909] Lustre: Skipped 1 previous similar message [ 8095.379751] Lustre: lustre-MDT0001: new disk, initializing [ 8095.462423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8095.502961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8095.517853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8100.844962] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8110.832920] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8117.305689] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8125.474974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8125.863974] Lustre: lustre-OST0000: new disk, initializing [ 8125.870732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8125.883742] Lustre: 187691:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8126.018764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8132.727182] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8135.249063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8135.260503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8135.422585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8142.390201] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8147.348297] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8147.535753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8152.508849] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8152.755846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8156.128485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8156.134669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8156.144854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8156.152865] Lustre: Skipped 2 previous similar messages [ 8156.163511] Lustre: Skipped 3 previous similar messages [ 8161.218961] Lustre: server umount lustre-OST0000 complete [ 8166.371970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8166.380579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8166.399323] Lustre: Skipped 1 previous similar message [ 8171.489899] LustreError: 185735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8171.493128] Lustre: server umount lustre-MDT0000 complete [ 8175.776762] LustreError: 187701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400586 with bad export cookie 8071444625943476212 [ 8175.780560] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8175.785070] LustreError: 187701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8176.268881] Lustre: server umount lustre-MDT0001 complete [ 8189.000213] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 08:09:57 (1787400597) [ 8208.658845] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8211.413418] Key type lgssc unregistered [ 8211.732741] LNet: 190638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8211.741877] LNetError: 190638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8212.777982] LNet: Removed LNI 192.168.202.118@tcp [ 8213.609203] Key type .llcrypt unregistered [ 8213.612080] Key type ._llcrypt unregistered [ 8231.936146] Key type ._llcrypt registered [ 8231.937714] Key type .llcrypt registered [ 8232.006000] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8233.148104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8233.180747] alg: No test for adler32 (adler32-zlib) [ 8234.300569] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 8234.646595] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8236.351310] Key type lgssc registered [ 8237.409749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8244.141660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8244.159279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8245.564717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8250.135230] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8254.945557] LustreError: 191860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8254.967261] LustreError: 191860:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8258.357706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8258.618980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8263.824865] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8270.859830] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8276.280375] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8286.124536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8286.729500] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8286.736598] LustreError: 193723:0:(llog.c:1696:llog_backup()) MGC192.168.202.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8286.752990] Lustre: 193723:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.118@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8287.017808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8296.209345] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8306.561787] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8309.731413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8309.744389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8309.761394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8313.829646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8313.852611] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.846012] Lustre: server umount lustre-MDT0000 complete [ 8318.951680] LustreError: 192826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8318.971572] LustreError: 192826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8320.194635] LustreError: 191841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400730 with bad export cookie 15783769100764255051 [ 8320.208712] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8320.208244] LustreError: 191841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8320.739377] Lustre: server umount lustre-MDT0001 complete [ 8335.310633] Lustre: server umount lustre-OST0000 complete [ 8345.118319] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 8356.520702] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8401.279766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8401.557567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8401.604416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8401.699671] Lustre: lustre-MDT0000: new disk, initializing [ 8401.807577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8401.832727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8408.220703] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8421.635046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8421.887412] Lustre: 197250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8421.964233] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8421.969931] Lustre: Skipped 1 previous similar message [ 8421.984659] LustreError: 198033:0:(llog.c:1696:llog_backup()) MGC192.168.202.118@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8421.994085] LustreError: 198033:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8422.005562] Lustre: 198033:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.118@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8422.049956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8422.057933] Lustre: Skipped 1 previous similar message [ 8422.147989] Lustre: lustre-MDT0001: new disk, initializing [ 8422.314597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8422.359456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8422.372473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8427.706591] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8438.404342] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8445.283586] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8452.268167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8452.533238] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8452.535837] Lustre: Skipped 1 previous similar message [ 8452.538404] LustreError: 199195:0:(llog.c:1696:llog_backup()) MGC192.168.202.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8452.545697] LustreError: 199195:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8452.554129] Lustre: 199195:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.118@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8452.564847] Lustre: 199195:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8452.599760] Lustre: lustre-OST0000: new disk, initializing [ 8452.605734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8452.612345] Lustre: 199218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8452.684292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8459.710206] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8461.853301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8461.865185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8461.978701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8469.173600] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8474.341170] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8474.724600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8479.656171] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8479.931657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8482.272371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8482.278697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8482.285498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8482.295143] Lustre: Skipped 2 previous similar messages [ 8482.305312] Lustre: Skipped 3 previous similar messages [ 8487.399398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8488.193342] Lustre: server umount lustre-OST0000 complete [ 8492.512986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8492.525990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8492.534474] Lustre: Skipped 2 previous similar messages [ 8498.480551] Lustre: server umount lustre-MDT0000 complete [ 8502.145274] LustreError: 198054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400912 with bad export cookie 15783769100764255772 [ 8502.146631] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8502.162457] LustreError: 198054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8502.654728] Lustre: server umount lustre-MDT0001 complete [ 8514.300640] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 08:15:22 (1787400922) [ 8540.243525] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8554.021299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8554.416971] LustreError: 202368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8554.492500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8559.208475] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8559.591229] LustreError: 202369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8564.705701] LustreError: 202368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8569.716698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8576.255202] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8580.728558] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8590.405762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8590.912070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8590.917448] Lustre: Skipped 1 previous similar message [ 8599.044332] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8611.252436] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8611.422400] Lustre: lustre-OST0001: new disk, initializing [ 8611.425706] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8611.430937] Lustre: 204681:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8617.067507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8617.080835] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8617.203447] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8618.101178] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8630.206734] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8638.427699] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8640.745917] Lustre: DEBUG MARKER: ADD OST3 [ 8646.692927] loop: module loaded [ 8646.776289] loop0: detected capacity change from 0 to 204800000 [ 8647.060372] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8647.067429] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8647.078534] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8653.835286] loop0: detected capacity change from 0 to 204800000 [ 8654.688728] Lustre: DEBUG MARKER: STOP OST3 [ 8660.552631] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8660.847613] Lustre: lustre-OST0003: new disk, initializing [ 8660.862993] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8660.878288] Lustre: 206673:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8661.022602] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8661.042123] Lustre: Skipped 1 previous similar message [ 8662.161027] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8662.359147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8662.373463] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8662.431181] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8664.231213] LustreError: 206777:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8665.608597] Lustre: Failing over lustre-OST0003 [ 8665.893591] Lustre: server umount lustre-OST0003 complete [ 8667.639843] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8667.652950] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8667.660448] Lustre: Skipped 1 previous similar message [ 8667.664958] LustreError: 203857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8667.714394] LustreError: 203857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8668.652828] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8678.460329] Lustre: DEBUG MARKER: START OST3 again [ 8686.769864] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8687.215682] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8688.477444] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8688.984552] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8688.986332] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8693.920423] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8717.295326] Lustre: Failing over lustre-OST0003 [ 8717.424687] Lustre: server umount lustre-OST0003 complete [ 8718.333537] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8718.338647] LustreError: 203859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8718.351961] Lustre: Skipped 2 previous similar messages [ 8718.381578] LustreError: 203859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 8738.789362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8738.793513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8738.801387] Lustre: Skipped 2 previous similar messages [ 8738.818525] Lustre: Skipped 5 previous similar messages [ 8744.151312] Lustre: server umount lustre-MDT0000 complete [ 8748.543131] LustreError: 202349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401158 with bad export cookie 15783769100764256654 [ 8748.557548] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8748.566517] LustreError: 202349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8753.057549] LustreError: 205478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8753.075263] LustreError: 205478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 8755.102462] Lustre: server umount lustre-MDT0001 complete [ 8760.135868] Lustre: server umount lustre-OST0000 complete [ 8764.609584] Lustre: server umount lustre-OST0001 complete [ 8773.125775] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 8783.511281] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8828.707949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8829.047221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8829.084557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8829.168786] Lustre: lustre-MDT0000: new disk, initializing [ 8829.282493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8829.292789] Lustre: Skipped 1 previous similar message [ 8829.317488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8833.599404] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8844.544873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8844.660662] Lustre: 212140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8844.979532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8849.722537] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8860.387350] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8867.849980] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8876.771594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8877.099835] Lustre: lustre-OST0000: new disk, initializing [ 8877.105958] Lustre: Skipped 1 previous similar message [ 8877.112204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8877.121153] Lustre: Skipped 2 previous similar messages [ 8877.125890] Lustre: 214110:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8878.322886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8878.340864] Lustre: Skipped 1 previous similar message [ 8878.345488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8878.524159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8884.369864] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8894.109624] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8899.537527] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8899.802834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8904.809272] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8905.145769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8909.281370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8909.282683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8909.303875] LustreError: Skipped 1 previous similar message [ 8909.314949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8909.346382] Lustre: Skipped 4 previous similar messages [ 8909.380694] Lustre: Skipped 9 previous similar messages [ 8913.450696] Lustre: server umount lustre-OST0000 complete [ 8924.116475] Lustre: server umount lustre-MDT0000 complete [ 8924.646847] LustreError: 213025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8924.679358] LustreError: 213025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8927.929764] LustreError: 212945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401338 with bad export cookie 15783769100764258950 [ 8927.942087] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8927.948988] LustreError: 212945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8928.464402] Lustre: server umount lustre-MDT0001 complete [ 8933.033978] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 08:22:21 (1787401341) [ 8954.884226] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 8971.394538] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9013.781214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9014.249779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9014.342880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9014.413327] Lustre: lustre-MDT0000: new disk, initializing [ 9014.543807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9021.231024] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9034.987084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9035.129230] Lustre: 218742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9035.392719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9040.598798] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9051.173620] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9057.935718] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9066.323817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9066.887459] Lustre: 220713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9068.868710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9069.068417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9076.026227] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9088.258965] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9094.066733] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9094.230770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9099.056468] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9099.271981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9104.864650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9104.868451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9104.872391] LustreError: Skipped 1 previous similar message [ 9104.875059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9104.896586] Lustre: Skipped 3 previous similar messages [ 9104.921180] Lustre: Skipped 3 previous similar messages [ 9108.057636] Lustre: server umount lustre-OST0000 complete [ 9118.792529] Lustre: server umount lustre-MDT0000 complete [ 9120.228382] LustreError: 218755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9120.246060] LustreError: 218755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9122.949546] LustreError: 218735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401533 with bad export cookie 15783769100764259832 [ 9122.978455] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9123.782626] Lustre: server umount lustre-MDT0001 complete [ 9133.654589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9134.245967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9134.254022] Lustre: Skipped 5 previous similar messages [ 9139.171589] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9149.430366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9155.336660] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9164.461379] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9171.124561] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9178.409908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9186.203364] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9198.211551] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9210.855140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9210.859436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9210.860808] Lustre: Skipped 2 previous similar messages [ 9215.006663] Lustre: server umount lustre-OST0000 complete [ 9225.976724] Lustre: server umount lustre-MDT0000 complete [ 9230.815324] LustreError: 222916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401641 with bad export cookie 15783769100764260714 [ 9230.819471] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9230.829510] LustreError: 222916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9231.232706] Lustre: server umount lustre-MDT0001 complete [ 9338.352116] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 9342.514855] Key type lgssc unregistered [ 9342.925049] LNet: 226598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9342.938177] LNetError: 226598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9342.978856] LNet: Removed LNI 192.168.202.118@tcp [ 9344.177315] Key type .llcrypt unregistered [ 9344.179252] Key type ._llcrypt unregistered [ 9372.636353] Key type ._llcrypt registered [ 9372.640649] Key type .llcrypt registered [ 9372.793564] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 9392.761582] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9394.178930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9394.214887] alg: No test for adler32 (adler32-zlib) [ 9395.173184] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 9395.591368] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 9397.256114] Key type lgssc registered [ 9398.661059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9439.235666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9439.267613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9440.639420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9440.672342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9440.804703] Lustre: lustre-MDT0000: new disk, initializing [ 9440.929986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9440.953073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9446.500514] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9457.971853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9458.187156] Lustre: 229955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9458.296258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9458.301102] Lustre: Skipped 1 previous similar message [ 9458.396989] Lustre: lustre-MDT0001: new disk, initializing [ 9458.501407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9458.583398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9458.598608] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9463.638953] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9475.282981] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9483.256986] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9490.510304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9490.865582] Lustre: lustre-OST0000: new disk, initializing [ 9490.869735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9490.877030] Lustre: 231924:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9490.971289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9496.662485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9496.685500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9496.815840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9497.359974] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9506.385644] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9512.478307] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9512.796723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9517.774083] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9517.938918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9522.663929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9522.671698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9522.679661] LustreError: Skipped 1 previous similar message [ 9522.680649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9522.718928] Lustre: Skipped 1 previous similar message [ 9526.359791] Lustre: server umount lustre-OST0000 complete [ 9530.339207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9530.355357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9530.371971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9530.377668] Lustre: Skipped 1 previous similar message [ 9532.897601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9532.897611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9536.262954] Lustre: server umount lustre-MDT0000 complete [ 9540.576054] LustreError: 230748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401950 with bad export cookie 15015548447219837939 [ 9540.584199] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9540.587335] LustreError: 230748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9540.964840] Lustre: server umount lustre-MDT0001 complete [ 9566.425873] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9579.291252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9579.809897] LustreError: 234879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9579.842060] LustreError: 234879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9579.893720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9584.838925] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9585.121523] LustreError: 234880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9590.251059] LustreError: 234879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9593.707310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9594.133269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9598.406739] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9601.132843] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9608.036904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9608.309928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9615.236632] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9625.368368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9625.570548] Lustre: lustre-OST0001: new disk, initializing [ 9625.578323] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9625.586317] Lustre: 237188:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9625.665398] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9631.360192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9631.384189] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9631.566505] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9632.115276] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9643.692586] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9649.653676] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9664.290488] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 08:34:32 (1787402072) [ 9667.558906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9667.560562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9667.582889] Lustre: Skipped 3 previous similar messages [ 9667.604271] Lustre: Skipped 4 previous similar messages [ 9672.673659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9672.680478] Lustre: Skipped 3 previous similar messages [ 9681.375189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9681.677928] Lustre: server umount lustre-MDT0000 complete [ 9682.953691] LustreError: 234875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9682.984528] LustreError: 234875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9685.733891] LustreError: 234861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402096 with bad export cookie 15015548447219838821 [ 9685.748720] LustreError: 234861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9685.750966] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9686.025476] Lustre: server umount lustre-MDT0001 complete [ 9700.539320] Lustre: server umount lustre-OST0000 complete [ 9704.418306] Lustre: 228178:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787402098/real 1787402098] req@ffff95c8473edc00 x1874226635193856/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787402114 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9704.436265] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9704.595785] Lustre: server umount lustre-OST0001 complete [ 9712.621578] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 9724.733127] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9771.068847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9771.366456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9771.391324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9771.452972] Lustre: lustre-MDT0000: new disk, initializing [ 9771.518161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9771.539027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9776.699703] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9788.773620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9788.910573] Lustre: 241138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9788.965612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9788.969307] Lustre: Skipped 1 previous similar message [ 9789.075543] Lustre: lustre-MDT0001: new disk, initializing [ 9789.267311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9789.281465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9793.839686] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9803.987474] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9812.534920] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9821.437378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9821.767602] Lustre: lustre-OST0000: new disk, initializing [ 9821.776208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9821.792482] Lustre: 243109:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9823.345468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9823.361767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9823.548184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9829.163882] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9841.201151] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9846.151963] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9846.296257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9850.887577] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9851.151697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9854.432558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9854.440695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9854.445749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9854.453448] Lustre: Skipped 1 previous similar message [ 9854.466214] Lustre: Skipped 5 previous similar messages [ 9859.422189] Lustre: server umount lustre-OST0000 complete [ 9869.483449] Lustre: server umount lustre-MDT0000 complete [ 9869.794194] LustreError: 241150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9869.823506] LustreError: 241150:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9873.157416] LustreError: 241131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402283 with bad export cookie 15015548447219840956 [ 9873.160582] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9873.167980] LustreError: 241131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9873.524672] Lustre: server umount lustre-MDT0001 complete [ 9883.002608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9883.466267] LustreError: 245328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9883.538822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9883.541710] Lustre: Skipped 2 previous similar messages [ 9888.055958] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9897.062696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9904.620039] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9912.783405] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9920.041260] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9928.396422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9936.297474] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9944.662171] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10092.513411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10092.523543] LustreError: Skipped 1 previous similar message [10092.530459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10092.553492] Lustre: Skipped 2 previous similar messages [10092.556150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10092.561777] Lustre: Skipped 2 previous similar messages [10105.311148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10105.404548] Lustre: server umount lustre-OST0000 complete [10112.480606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10116.260295] Lustre: server umount lustre-MDT0000 complete [10119.137143] LustreError: 245328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10119.165769] LustreError: 245328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10120.297905] LustreError: 246065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402530 with bad export cookie 15015548447219841838 [10120.309065] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10120.316903] LustreError: 246065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10120.715065] Lustre: server umount lustre-MDT0001 complete [10216.301656] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [10219.527523] Key type lgssc unregistered [10219.996988] LNet: 248937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10220.007486] LNetError: 248937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10220.041941] LNet: Removed LNI 192.168.202.118@tcp [10221.353038] Key type .llcrypt unregistered [10221.359629] Key type ._llcrypt unregistered [10253.543222] Key type ._llcrypt registered [10253.545511] Key type .llcrypt registered [10253.658284] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [10273.460517] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [10274.559980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10274.644046] alg: No test for adler32 (adler32-zlib) [10275.870387] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [10276.161610] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [10277.831263] Key type lgssc registered [10278.976080] Lustre: Echo OBD driver; http://www.lustre.org/ [10334.468388] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [10346.840318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10346.893685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10348.247376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10348.301074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10348.420801] Lustre: lustre-MDT0000: new disk, initializing [10348.512233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10348.541492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10352.882659] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10363.867465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10364.040837] Lustre: 253021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10364.133807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10364.137706] Lustre: Skipped 1 previous similar message [10364.231550] Lustre: lustre-MDT0001: new disk, initializing [10364.377547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10364.418694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10364.445301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10368.572693] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10374.054217] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10381.970888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10382.185339] Lustre: lustre-OST0000: new disk, initializing [10382.189246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10382.194873] Lustre: 254653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10382.298697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10387.569872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10387.578412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10387.737395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10389.059380] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10400.791932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10400.961717] Lustre: lustre-OST0001: new disk, initializing [10400.964870] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10400.969843] Lustre: 255526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10401.062758] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10408.279596] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10409.067978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10409.099537] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10409.213523] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10420.681655] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10427.324903] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10440.168789] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10440.172125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10440.186304] Lustre: Skipped 3 previous similar messages [10440.208371] Lustre: Skipped 3 previous similar messages [10445.290423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10445.303793] Lustre: Skipped 3 previous similar messages [10446.670920] Lustre: server umount lustre-MDT0000 complete [10450.239330] LustreError: 253012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402860 with bad export cookie 15293669907877520758 [10450.246535] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10450.249444] LustreError: 253012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10450.409405] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10450.426405] Lustre: Skipped 1 previous similar message [10450.428752] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10450.437776] Lustre: Skipped 1 previous similar message [10450.820603] Lustre: server umount lustre-MDT0001 complete [10456.811725] Lustre: server umount lustre-OST0000 complete [10462.147710] Lustre: server umount lustre-OST0001 complete [10480.575344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10491.367648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10501.668483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10512.094577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10535.412357] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [10549.864522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10549.880280] Lustre: lustre-MDT0000: reset Object Index mappings [10550.604622] LustreError: 260535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10550.654924] LustreError: 260535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10550.754847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10555.872585] LustreError: 260536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10556.179540] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10560.012189] LustreError: 260535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10565.094564] LustreError: 260536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10566.012645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10566.030925] Lustre: lustre-MDT0001: reset Object Index mappings [10566.432603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10571.509132] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10574.687480] Lustre: 261678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10583.146824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10583.760653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10590.953115] LustreError: 262033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10591.292299] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10595.092439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10600.430521] LustreError: 262033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10600.457112] LustreError: 262033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10602.448994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10608.128365] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10609.745920] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10619.320201] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10623.422164] Lustre: 263548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10629.331589] Lustre: server umount lustre-MDT0000 complete [10633.190930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10633.199832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10633.221479] LustreError: 260535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10633.241379] LustreError: 260516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403043 with bad export cookie 15293669907877527548 [10633.245719] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10633.624105] Lustre: server umount lustre-MDT0001 complete [10649.567318] Lustre: 250525:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787403043/real 1787403043] req@ffff95c7463d8e00 x1874227559004032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787403059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10649.609937] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10654.751552] Lustre: 250525:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787403049/real 1787403049] req@ffff95c7463d9f80 x1874227559004544/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787403065 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10654.792702] Lustre: 250525:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10657.962649] Lustre: server umount lustre-OST0000 complete [10658.783106] Lustre: 250523:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787403053/real 1787403053] req@ffff95c743e78e00 x1874227559005056/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787403069 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10658.813515] Lustre: 250523:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10663.011863] Lustre: server umount lustre-OST0001 complete [10685.977895] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 08:51:34 (1787403094) [10704.812647] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [10719.100392] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [10761.422198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10761.780251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10761.803858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10761.906555] Lustre: lustre-MDT0000: new disk, initializing [10762.009255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10762.020756] Lustre: Skipped 1 previous similar message [10762.046349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10767.621712] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10780.750179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10780.898766] Lustre: 267817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10780.956210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10780.960219] Lustre: Skipped 1 previous similar message [10781.027097] Lustre: lustre-MDT0001: new disk, initializing [10781.144683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10781.154951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10786.489996] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10797.831910] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10804.105836] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10811.233805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10811.564644] Lustre: lustre-OST0000: new disk, initializing [10811.574493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10811.589883] Lustre: 269789:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10813.680088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10813.699408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10813.847842] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10819.252490] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10832.271507] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10838.525706] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10838.704669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10844.241637] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10844.423375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10849.767937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10849.780081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10849.798096] Lustre: Skipped 1 previous similar message [10849.805886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10849.808180] LustreError: Skipped 1 previous similar message [10852.756548] Lustre: server umount lustre-OST0000 complete [10857.957217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10857.963434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10857.998800] Lustre: Skipped 1 previous similar message [10858.006038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10858.012532] Lustre: Skipped 1 previous similar message [10863.643890] Lustre: server umount lustre-MDT0000 complete [10868.154483] LustreError: 267812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403278 with bad export cookie 15293669907877529634 [10868.164750] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10868.504887] Lustre: server umount lustre-MDT0001 complete [10890.132253] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [10902.526205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10903.007555] LustreError: 272726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10903.110196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10903.119310] Lustre: Skipped 2 previous similar messages [10908.457120] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10919.069868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10925.012243] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10928.793861] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10938.882041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10948.053158] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10958.226362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10958.399278] Lustre: lustre-OST0001: new disk, initializing [10958.404306] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10958.408862] Lustre: 275038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10959.888072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10959.901803] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10960.053144] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10966.065769] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10977.424966] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10981.132419] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10987.512922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10987.694426] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11000.801262] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11000.817417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11000.829409] Lustre: Skipped 3 previous similar messages [11000.841439] Lustre: Skipped 4 previous similar messages [11002.817745] Lustre: server umount lustre-MDT0000 complete [11005.923811] LustreError: 272720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11005.971277] LustreError: 272720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [11006.975620] LustreError: 272708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403417 with bad export cookie 15293669907877530523 [11006.975959] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11006.987036] LustreError: 272708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11007.331627] Lustre: server umount lustre-MDT0001 complete [11021.683586] Lustre: server umount lustre-OST0000 complete [11036.708574] Lustre: server umount lustre-OST0001 complete [11058.760598] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [11071.513826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11072.176329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11072.179560] Lustre: Skipped 3 previous similar messages [11077.337666] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11087.100944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11092.503914] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11095.780030] Lustre: 279238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11102.849778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11109.956794] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11118.581809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11119.955635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11125.783465] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11127.494349] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11137.033907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11141.770604] Lustre: 281115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11147.704977] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11147.790059] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11159.026125] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 08:59:27 (1787403567) [11174.883595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11174.890504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11174.906869] Lustre: Skipped 1 previous similar message [11174.919267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11174.924034] Lustre: Skipped 1 previous similar message [11180.277416] Lustre: server umount lustre-MDT0000 complete [11182.055854] LustreError: 279614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11182.073991] LustreError: 279614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [11183.982054] LustreError: 279240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403594 with bad export cookie 15293669907877537054 [11183.990432] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11183.997406] LustreError: 279240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11190.418122] Lustre: server umount lustre-MDT0001 complete [11200.710837] Lustre: server umount lustre-OST0000 complete [11204.849702] Lustre: server umount lustre-OST0001 complete [11213.270074] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [11225.957561] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [11287.143142] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [11299.984783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11300.375872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11300.415240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11300.481270] Lustre: lustre-MDT0000: new disk, initializing [11300.562149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11305.737534] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11318.202119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11318.340329] Lustre: 285393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11318.648281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11323.692752] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11336.235344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11336.587442] Lustre: 287026:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11336.796218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11336.804990] Lustre: Skipped 5 previous similar messages [11338.388351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11343.478326] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11355.657477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11355.831264] Lustre: 287897:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11357.804237] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11357.819418] Lustre: Skipped 1 previous similar message [11357.909580] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11363.366326] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11373.621797] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11377.383616] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11377.393837] Lustre: Skipped 1 previous similar message [11394.896770] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:03:23 (1787403803) [11399.139968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11399.153754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11399.169093] Lustre: Skipped 8 previous similar messages [11399.177293] Lustre: Skipped 10 previous similar messages [11399.529793] Lustre: server umount lustre-MDT0000 complete [11403.215376] LustreError: 286189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403813 with bad export cookie 15293669907877544705 [11403.226149] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11403.227651] LustreError: 286189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11403.551825] Lustre: server umount lustre-MDT0001 complete [11418.603103] Lustre: server umount lustre-OST0000 complete [11420.323044] Lustre: 250526:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787403814/real 1787403814] req@ffff95c749429c00 x1874227559277568/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787403830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11420.361247] Lustre: 250526:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11424.068686] Lustre: server umount lustre-OST0001 complete [11433.879695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11438.446967] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11439.586767] LustreError: 290143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11439.600072] LustreError: 290143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11447.338686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11452.678430] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11460.372521] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11467.382670] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11475.549367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11482.046645] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11489.700379] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11509.254849] loop1: detected capacity change from 0 to 204800000 [11509.545346] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11509.553155] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11509.568391] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11514.988771] loop1: detected capacity change from 0 to 204800000 [11520.458309] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11520.542932] LustreError: 290131:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs24724: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11520.551624] LustreError: 290131:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs24724 [11520.555874] LustreError: 290131:0:(mgs_handler.c:517:mgs_target_reg()) fs24724-MDT0001: Error checking target index: rc = -13 [11520.565467] LustreError: fs24724-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11520.573795] LustreError: 293859:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11520.579146] LustreError: 293859:0:(tgt_mount.c:2129:server_put_super()) no obd fs24724-MDT0001 [11520.583386] LustreError: 293859:0:(tgt_mount.c:116:server_deregister_mount()) fs24724-MDT0001 not registered [11520.608584] Lustre: server umount fs24724-MDT0001 complete [11520.612160] LustreError: 293859:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11528.561366] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11528.668793] Lustre: Setting parameter fs24724-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs24724-MDT0001 [11528.720338] Lustre: srv-fs24724-MDT0001: No data found on store. Initialize space. [11528.728059] Lustre: Skipped 4 previous similar messages [11528.823670] Lustre: fs24724-MDT0001: new disk, initializing [11528.827549] Lustre: Skipped 3 previous similar messages [11528.954104] Lustre: srv-fs24724-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11528.964519] Lustre: 294327:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24724-MDT0001-mdtlov: can't create new llog: rc = -115 [11528.978832] LustreError: 294327:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24724-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11533.801722] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11541.377619] Lustre: server umount fs24724-MDT0001 complete [11553.965018] loop1: detected capacity change from 0 to 204800000 [11558.887168] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11559.315656] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11559.729591] Lustre: 295895:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24724-MDT0001-mdtlov: can't create new llog: rc = -115 [11559.744787] LustreError: 295895:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24724-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [11564.926474] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11570.147411] Lustre: server umount fs24724-MDT0001 complete [11574.263589] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11580.901523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11580.914284] Lustre: Skipped 1 previous similar message [11580.921635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11580.926261] Lustre: Skipped 1 previous similar message [11586.589397] Lustre: server umount lustre-MDT0000 complete [11594.563996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11594.676646] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11599.610093] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11600.378281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11607.937990] loop1: detected capacity change from 0 to 204800000 [11613.302968] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11613.813773] Lustre: 298341:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24724-MDT0001-mdtlov: can't create new llog: rc = -115 [11613.831659] LustreError: 298341:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24724-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11618.842402] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11624.650304] Lustre: server umount fs24724-MDT0001 complete [11656.160533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11656.166215] LustreError: Skipped 2 previous similar messages [11659.662429] Lustre: server umount lustre-OST0000 complete [11670.971354] Lustre: server umount lustre-MDT0000 complete [11676.411280] LustreError: 294872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404086 with bad export cookie 15293669907877547386 [11676.419184] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11676.438310] LustreError: 294872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11677.218586] Lustre: server umount lustre-MDT0001 complete [11691.271676] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [11694.166411] Key type lgssc unregistered [11694.388787] LNet: 300685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11694.395851] LNetError: 300685:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11694.409821] LNet: Removed LNI 192.168.202.118@tcp [11695.130270] Key type .llcrypt unregistered [11695.132083] Key type ._llcrypt unregistered [11715.784748] Key type ._llcrypt registered [11715.788301] Key type .llcrypt registered [11715.902302] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [11717.439070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11717.492549] alg: No test for adler32 (adler32-zlib) [11718.610625] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [11718.840645] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [11720.584907] Key type lgssc registered [11721.735869] Lustre: Echo OBD driver; http://www.lustre.org/ [11731.005523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11731.031738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11732.411703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11737.319830] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11742.693058] LustreError: 301958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11742.716662] LustreError: 301958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11746.303360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11746.988900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11752.293489] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11760.582765] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11766.624986] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11773.835902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11774.277450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11777.385085] LustreError: 303786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11780.678443] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11782.650156] LustreError: 303788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11787.750867] LustreError: 303787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11788.998481] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11791.780412] LustreError: 303787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11802.021131] LustreError: 303786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11802.057160] LustreError: 303786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11811.153818] loop1: detected capacity change from 0 to 204800000 [11811.431741] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11811.452329] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11811.468805] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11817.217029] loop1: detected capacity change from 0 to 204800000 [11822.503328] LustreError: 303788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11822.530083] LustreError: 303788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [11823.517653] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11823.653781] Lustre: fs12817-OST0000: new disk, initializing [11823.657511] Lustre: srv-fs12817-OST0000: No data found on store. Initialize space. [11823.663697] Lustre: 305691:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11823.720183] Lustre: fs12817-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11829.842261] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11837.274086] Lustre: server umount fs12817-OST0000 complete [11849.596471] loop1: detected capacity change from 0 to 204800000 [11854.083888] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11854.252979] Lustre: fs12817-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11858.338703] LustreError: 303788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11858.367837] LustreError: 303788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [11858.410755] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11860.430475] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11868.818474] loop2: detected capacity change from 0 to 204800000 [11869.106215] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11869.130504] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11869.158853] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11873.788973] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11874.934476] loop2: detected capacity change from 0 to 204800000 [11879.854771] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11879.969686] LustreError: fs12817-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11879.980537] LustreError: fs12817-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11879.994091] LustreError: 308733:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11880.000844] LustreError: 308733:0:(tgt_mount.c:2129:server_put_super()) no obd fs12817-OST0001 [11880.003933] LustreError: 308733:0:(tgt_mount.c:116:server_deregister_mount()) fs12817-OST0001 not registered [11880.029633] Lustre: server umount fs12817-OST0001 complete [11880.031606] LustreError: 308733:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11890.298680] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11890.709702] Lustre: fs12817-OST0001: new disk, initializing [11890.714604] Lustre: srv-fs12817-OST0001: No data found on store. Initialize space. [11890.724679] Lustre: 309285:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11890.837033] Lustre: fs12817-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11897.249783] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11905.076678] Lustre: server umount fs12817-OST0000 complete [11913.692607] Lustre: server umount fs12817-OST0001 complete [11924.899290] LustreError: 303788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11924.921958] LustreError: 303788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11926.543280] loop1: detected capacity change from 0 to 204800000 [11933.606161] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11933.979476] Lustre: fs12817-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11940.929827] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11948.221740] loop2: detected capacity change from 0 to 204800000 [11954.355438] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11961.625917] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11971.999092] Lustre: server umount fs12817-OST0000 complete [11982.680447] Lustre: server umount fs12817-OST0001 complete [11997.665701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11997.677902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11997.698892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12001.760947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12001.766251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12001.797873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12003.545693] Lustre: server umount lustre-OST0000 complete [12007.907903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12007.921931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12007.933964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12007.936055] Lustre: Skipped 1 previous similar message [12012.003157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12012.004399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12013.936348] Lustre: server umount lustre-MDT0000 complete [12017.577869] LustreError: 301938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404427 with bad export cookie 6048500831193577137 [12017.585353] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12017.595834] LustreError: 301938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12018.061945] Lustre: server umount lustre-MDT0001 complete [12029.665580] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12032.565525] Key type lgssc unregistered [12032.834838] LNet: 315207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12032.847134] LNetError: 315207:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12032.860279] LNet: Removed LNI 192.168.202.118@tcp [12033.659149] Key type .llcrypt unregistered [12033.663458] Key type ._llcrypt unregistered [12067.281428] Key type ._llcrypt registered [12067.283059] Key type .llcrypt registered [12067.508956] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [12084.418389] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12085.548918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12085.640608] alg: No test for adler32 (adler32-zlib) [12086.664759] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [12086.894717] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12088.633788] Key type lgssc registered [12089.831707] Lustre: Echo OBD driver; http://www.lustre.org/ [12130.372193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12130.411504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12131.670447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12131.731351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12131.837185] Lustre: lustre-MDT0000: new disk, initializing [12131.976875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12131.998404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12137.195194] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12149.568656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12149.797784] Lustre: 319067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12149.865644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12149.875520] Lustre: Skipped 1 previous similar message [12150.029828] Lustre: lustre-MDT0001: new disk, initializing [12150.169043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12150.225714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12150.237078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12155.855888] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12167.604430] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12174.489731] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12182.383529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12182.768842] Lustre: lustre-OST0000: new disk, initializing [12182.775932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12182.787246] Lustre: 321040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12182.955387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12190.309714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12190.325578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12190.504898] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12190.539159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12201.466869] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12203.735810] LustreError: 317242:0:(lmv_obd.c:208:lmv_notify()) activation of lustre-MDT0001_UUID failed: -22 [12203.792617] Lustre: Mounted lustre-client [12205.088206] Lustre: Unmounted lustre-client [12216.288576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12216.297160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12216.323685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12216.339594] Lustre: Skipped 1 previous similar message [12219.375582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12219.396197] Lustre: Skipped 1 previous similar message [12222.278384] Lustre: server umount lustre-OST0000 complete [12227.041301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12227.049326] LustreError: Skipped 1 previous similar message [12227.054052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12227.065016] Lustre: Skipped 1 previous similar message [12227.069595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12229.603264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12229.603613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12234.722040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12234.733483] Lustre: Skipped 2 previous similar messages [12240.351143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12240.571773] Lustre: server umount lustre-MDT0000 complete [12244.824370] LustreError: 321050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404655 with bad export cookie 17229156310663702775 [12244.846282] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12244.850970] LustreError: 321050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12245.257384] Lustre: server umount lustre-MDT0001 complete [12255.970953] LNet: 1 local NIs in recovery (showing 1): 192.168.202.118@tcp [12257.340309] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12260.188205] Key type lgssc unregistered [12260.535936] LNet: 323202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12260.548039] LNetError: 323202:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12334.310442] LNet: Removed LNI 192.168.202.118@tcp [12335.032190] Key type .llcrypt unregistered [12335.034414] Key type ._llcrypt unregistered [12352.424905] Key type ._llcrypt registered [12352.427376] Key type .llcrypt registered [12352.515166] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [12367.917192] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12368.957178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12369.031030] alg: No test for adler32 (adler32-zlib) [12370.112620] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [12370.374632] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12372.071181] Key type lgssc registered [12373.280974] Lustre: Echo OBD driver; http://www.lustre.org/ [12407.690437] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 09:20:16 (1787404816) [12416.381078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12416.422659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12417.692148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12417.720650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12417.793763] Lustre: lustre-MDT0000: new disk, initializing [12417.861155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12417.876785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12422.882747] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12434.292602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12434.518392] Lustre: 326423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12434.649345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12434.659601] Lustre: Skipped 1 previous similar message [12434.832488] Lustre: lustre-MDT0001: new disk, initializing [12434.992499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12435.063208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12435.093746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12440.246232] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12450.793465] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12457.063419] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12464.242113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12464.541297] Lustre: lustre-OST0000: new disk, initializing [12464.546207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12464.557795] Lustre: 328392:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12464.670732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12470.377711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12470.387846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12470.516786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12471.285593] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12481.749785] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12501.504471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12501.504802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12501.508773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12501.521192] Lustre: Skipped 1 previous similar message [12501.556502] LustreError: Skipped 1 previous similar message [12503.732619] Lustre: server umount lustre-OST0000 complete [12511.713213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12511.718732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12511.723516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12511.738389] Lustre: Skipped 2 previous similar messages [12513.890848] Lustre: server umount lustre-MDT0000 complete [12517.407648] LustreError: 326413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404927 with bad export cookie 5610283564778734171 [12517.411321] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12517.414218] LustreError: 326413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12517.714318] Lustre: server umount lustre-MDT0001 complete [12532.674423] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12535.105577] Key type lgssc unregistered [12535.400086] LNet: 330302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12535.409524] LNetError: 330302:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12535.429217] LNet: Removed LNI 192.168.202.118@tcp [12536.342440] Key type .llcrypt unregistered [12536.344667] Key type ._llcrypt unregistered [12558.583108] Key type ._llcrypt registered [12558.585483] Key type .llcrypt registered [12558.658508] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12559.932608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12560.037175] alg: No test for adler32 (adler32-zlib) [12561.279429] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [12561.575903] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12563.303206] Key type lgssc registered [12564.758722] Lustre: Echo OBD driver; http://www.lustre.org/ [12572.254118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12572.284607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12573.801718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12578.730185] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12583.906626] LustreError: 331581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12583.936263] LustreError: 331581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12588.388734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12588.726034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12593.198394] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12601.748600] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12607.276317] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12613.762051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12614.045931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12619.501077] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12627.036868] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12640.053349] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12642.144691] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12648.937346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12648.940346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12648.959090] Lustre: Skipped 2 previous similar messages [12648.991582] Lustre: Skipped 2 previous similar messages [12652.693240] Lustre: server umount lustre-MDT0000 complete [12654.056397] LustreError: 331576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12654.097373] LustreError: 331576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12657.851822] LustreError: 331561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405068 with bad export cookie 3256930336045809938 [12657.859570] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12657.860839] LustreError: 331561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12659.168382] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12659.171884] LustreError: 331578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12659.191310] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12659.228888] LustreError: 331578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12659.246292] Lustre: Skipped 1 previous similar message [12662.182080] LustreError: 331578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12662.182646] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12662.201286] LustreError: 331578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12662.218221] Lustre: Skipped 1 previous similar message [12664.339702] Lustre: server umount lustre-MDT0001 complete [12668.795190] Lustre: server umount lustre-OST0000 complete [12678.580158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12684.020417] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12687.144482] Lustre: 335763:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.118@tcp [12689.296176] Lustre: server umount MGS complete [12700.559838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12701.101351] LustreError: 336335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12701.192385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12706.087405] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12711.411257] LustreError: 336336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12711.459353] LustreError: 336336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12716.673705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12717.235247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12722.177439] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12732.036794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12732.530807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12739.626577] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12747.642938] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12749.572985] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12756.657806] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12758.773798] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12765.167621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12765.180202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12765.204859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12766.285496] Lustre: server umount lustre-OST0000 complete [12772.832625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12772.839294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12772.857044] Lustre: Skipped 1 previous similar message [12772.876032] Lustre: Skipped 1 previous similar message [12783.071297] Lustre: 330894:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787405177/real 1787405177] req@0000000007cf566f x1874229954934272/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787405193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12783.077425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12783.129924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12783.133028] Lustre: Skipped 3 previous similar messages [12785.119593] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12785.252333] Lustre: 330894:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787405179/real 1787405179] req@00000000853ded3a x1874229954934784/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787405195 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12785.455210] Lustre: server umount lustre-MDT0000 complete [12788.192775] LustreError: 336332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12788.223412] LustreError: 336332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12789.417872] Lustre: 330894:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787405183/real 1787405183] req@000000003ff83395 x1874229954936064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787405199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12789.863298] LustreError: 337791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405200 with bad export cookie 3256930336045810778 [12789.873074] LustreError: 337791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12789.881694] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12790.350669] Lustre: server umount lustre-MDT0001 complete [12802.472238] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12805.267578] Key type lgssc unregistered [12805.709762] LNet: 339770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12805.724564] LNetError: 339770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12805.756261] LNet: Removed LNI 192.168.202.118@tcp [12806.851377] Key type .llcrypt unregistered [12806.852923] Key type ._llcrypt unregistered [12827.733278] Key type ._llcrypt registered [12827.735045] Key type .llcrypt registered [12827.944489] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [12848.448279] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12849.827558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12849.876088] alg: No test for adler32 (adler32-zlib) [12850.968642] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [12851.255662] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12852.959600] Key type lgssc registered [12854.451416] Lustre: Echo OBD driver; http://www.lustre.org/ [12892.075868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12892.095304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12893.400880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12893.431462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12893.512026] Lustre: lustre-MDT0000: new disk, initializing [12893.614266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12893.628840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12897.755354] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12909.441170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12909.550228] Lustre: 342935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12909.602796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12909.605833] Lustre: Skipped 1 previous similar message [12909.670245] Lustre: lustre-MDT0001: new disk, initializing [12909.775717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12909.809358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12909.818411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12914.754809] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12926.677310] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12933.634937] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12941.586872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12941.909709] Lustre: lustre-OST0000: new disk, initializing [12941.913684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12941.921388] Lustre: 344905:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12942.050637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12949.094218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12949.115513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12949.273543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12949.907761] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12962.298711] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12968.567335] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12968.830725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12974.820067] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12975.025912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12980.193598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12980.205235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12980.220500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12980.231139] Lustre: Skipped 1 previous similar message [12983.626975] Lustre: server umount lustre-OST0000 complete [12990.433129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12990.449673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12990.450048] Lustre: Skipped 2 previous similar messages [12990.469243] Lustre: Skipped 1 previous similar message [12994.885273] Lustre: server umount lustre-MDT0000 complete [12999.929899] LustreError: 342926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405410 with bad export cookie 721956191602698916 [12999.934732] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12999.944792] LustreError: 342926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13000.591134] Lustre: server umount lustre-MDT0001 complete [13012.868197] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 09:30:21 (1787405421) [13032.344725] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [13043.582421] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13082.142047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13082.347485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13082.366810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13082.435327] Lustre: lustre-MDT0000: new disk, initializing [13082.536240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13082.587653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13087.974234] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13099.437294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13099.602881] Lustre: 349668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13099.664861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13099.667644] Lustre: Skipped 1 previous similar message [13099.776945] Lustre: lustre-MDT0001: new disk, initializing [13099.914503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13099.945582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13099.961897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13106.277854] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13117.235975] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13123.061734] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13130.203714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13130.488265] Lustre: lustre-OST0000: new disk, initializing [13130.491952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13130.500972] Lustre: 351636:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13130.629698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13137.835429] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13138.514097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13138.529283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13138.676176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13148.453405] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13195.241374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13195.243235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13195.246669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13195.267442] LustreError: Skipped 2 previous similar messages [13195.320878] Lustre: Skipped 1 previous similar message [13200.356114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13200.358730] Lustre: Skipped 2 previous similar messages [13200.973864] Lustre: server umount lustre-OST0000 complete [13205.472994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13205.477399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13205.492888] Lustre: Skipped 1 previous similar message [13205.503361] Lustre: Skipped 1 previous similar message [13207.650353] Lustre: server umount lustre-MDT0000 complete [13210.594601] LustreError: 350831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13210.619109] LustreError: 350831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13212.155345] LustreError: 349659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405622 with bad export cookie 721956191602699798 [13212.168985] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13212.668470] Lustre: server umount lustre-MDT0001 complete [13225.217748] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13228.497579] Key type lgssc unregistered [13228.834534] LNet: 353815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13228.840290] LNetError: 353815:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13228.858991] LNet: Removed LNI 192.168.202.118@tcp [13229.819478] Key type .llcrypt unregistered [13229.825084] Key type ._llcrypt unregistered [13328.271379] Key type ._llcrypt registered [13328.276258] Key type .llcrypt registered [13328.379526] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13328.628430] Key type .llcrypt unregistered [13328.635696] Key type ._llcrypt unregistered [13410.705150] Key type ._llcrypt registered [13410.707506] Key type .llcrypt registered [13410.827118] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13411.192840] Key type .llcrypt unregistered [13411.197985] Key type ._llcrypt unregistered [13448.217617] Key type ._llcrypt registered [13448.219777] Key type .llcrypt registered [13448.304950] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13448.522785] Key type .llcrypt unregistered [13448.524576] Key type ._llcrypt unregistered [13482.134369] Key type ._llcrypt registered [13482.136385] Key type .llcrypt registered [13482.212410] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13482.413608] Key type .llcrypt unregistered [13482.415896] Key type ._llcrypt unregistered [13565.323880] Key type ._llcrypt registered [13565.328474] Key type .llcrypt registered [13565.536724] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13565.843328] Key type .llcrypt unregistered [13565.845063] Key type ._llcrypt unregistered [13600.245622] Key type ._llcrypt registered [13600.250483] Key type .llcrypt registered [13600.382715] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13600.625373] Key type .llcrypt unregistered [13600.627433] Key type ._llcrypt unregistered [13637.320976] Key type ._llcrypt registered [13637.324664] Key type .llcrypt registered [13637.503904] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13637.892510] Key type .llcrypt unregistered [13637.894380] Key type ._llcrypt unregistered [13677.891388] Key type ._llcrypt registered [13677.892913] Key type .llcrypt registered [13677.976189] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13678.417312] Key type .llcrypt unregistered [13678.418944] Key type ._llcrypt unregistered [13712.208272] Key type ._llcrypt registered [13712.211376] Key type .llcrypt registered [13712.361536] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13712.688303] Key type .llcrypt unregistered [13712.690128] Key type ._llcrypt unregistered [13748.358186] Key type ._llcrypt registered [13748.361415] Key type .llcrypt registered [13748.534416] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13748.865935] Key type .llcrypt unregistered [13748.869102] Key type ._llcrypt unregistered [13817.538398] Key type ._llcrypt registered [13817.539848] Key type .llcrypt registered [13817.646466] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13817.939666] Key type .llcrypt unregistered [13817.942593] Key type ._llcrypt unregistered [13858.228319] Key type ._llcrypt registered [13858.231471] Key type .llcrypt registered [13858.367671] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13858.710491] Key type .llcrypt unregistered [13858.712628] Key type ._llcrypt unregistered [13893.997497] Key type ._llcrypt registered [13894.000075] Key type .llcrypt registered [13894.128095] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13894.471582] Key type .llcrypt unregistered [13894.477417] Key type ._llcrypt unregistered [13949.857713] Key type ._llcrypt registered [13949.860461] Key type .llcrypt registered [13950.003504] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [13965.762825] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13966.998257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13967.051074] alg: No test for adler32 (adler32-zlib) [13968.051745] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [13968.280280] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [13969.967275] Key type lgssc registered [13971.517450] Lustre: Echo OBD driver; http://www.lustre.org/ [14011.187647] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14011.261602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14012.623473] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14012.670479] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14012.764772] Lustre: tst250fs-MDT0000: new disk, initializing [14012.899045] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14012.931068] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14017.621131] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14029.140368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14029.265830] Lustre: 374201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14029.324168] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14029.327681] Lustre: Skipped 1 previous similar message [14029.386898] Lustre: tst250fs-MDT0001: new disk, initializing [14029.463876] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14029.506800] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14029.522293] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14034.426748] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14040.138310] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14048.456885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14048.815373] Lustre: tst250fs-OST0000: new disk, initializing [14048.820972] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14048.847251] Lustre: 375835:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14048.993737] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14050.637017] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14050.653064] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14050.864088] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14057.235708] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14067.999884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14068.157866] Lustre: tst250fs-OST0001: new disk, initializing [14068.160987] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14068.167795] Lustre: 376704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14068.267469] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14075.590428] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14077.584170] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14077.592106] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14077.657990] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14081.308709] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14084.003867] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14084.008763] Lustre: Skipped 2 previous similar messages [14106.607798] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14106.620147] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14106.642621] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14108.647805] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14108.670565] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14108.694075] Lustre: Skipped 1 previous similar message [14113.763070] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14113.780164] Lustre: Skipped 5 previous similar messages [14118.367591] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14118.706750] Lustre: server umount tst250fs-MDT0000 complete [14122.950378] LustreError: 374193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787406533 with bad export cookie 15987639621219677887 [14122.955984] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14122.960541] LustreError: 374193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14123.329360] Lustre: server umount tst250fs-MDT0001 complete [14138.185706] Lustre: server umount tst250fs-OST0000 complete [14140.394155] Lustre: 372423:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787406534/real 1787406534] req@ffff95c745b6dc00 x1874231430292864/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787406550 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14140.438406] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14143.037798] Lustre: server umount tst250fs-OST0001 complete [14163.279572] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [14172.574858] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [14212.514897] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 09:50:20 (1787406620) [14214.722844] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14235.910202] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [14247.184735] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [14284.958558] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13984 sec ========= 09:51:33 (1787406693) [14287.334887] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:51:35 (1787406695) === [14292.024592] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:51:40 (1787406700) === [14339.548573] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [14343.442342] Key type lgssc unregistered [14343.836312] LNet: 385859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14343.840085] LNetError: 385859:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14343.866806] LNet: Removed LNI 192.168.202.118@tcp [14344.870206] Key type .llcrypt unregistered [14344.872230] Key type ._llcrypt unregistered