[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 475194507 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002377] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007318] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009012] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011124] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014072] *** VALIDATE selinux *** [ 0.022519] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027125] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] *** VALIDATE tmpfs *** [ 0.032097] *** VALIDATE proc *** [ 0.033253] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035272] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042280] debug: unmapping init [mem 0xffffffff87059000-0xffffffff87060fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045723] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.053294] rcu: Hierarchical SRCU implementation. [ 0.055429] smp: Bringing up secondary CPUs ... [ 0.056634] x86: Booting SMP configuration: [ 0.057031] .... node #0, CPUs: #1 #2 #3 [ 0.060241] smp: Brought up 1 node, 4 CPUs [ 0.062011] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145023] node 0 deferred pages initialised in 80ms [ 0.150010] devtmpfs: initialized [ 0.151291] x86/mm: Memory block size: 128MB [ 0.153878] gcov: version magic: 0x41383552 [ 0.155295] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157263] pinctrl core: initialized pinctrl subsystem [ 0.158209] [ 0.158608] ************************************************************* [ 0.159018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160017] ** ** [ 0.161016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162015] ** ** [ 0.163017] ** This means that this kernel is built to expose internal ** [ 0.164018] ** IOMMU data structures, which may compromise security on ** [ 0.165017] ** your system. ** [ 0.166014] ** ** [ 0.167013] ** If you see this message and you are not debugging the ** [ 0.168015] ** kernel, report this immediately to your vendor! ** [ 0.169011] ** ** [ 0.170017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171014] ************************************************************* [ 0.172645] NET: Registered protocol family 16 [ 0.173452] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176415] cpuidle: using governor menu [ 0.178865] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182460] PCI: Using configuration type 1 for base access [ 0.184125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.194023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.196031] cryptd: max_cpu_qlen set to 1000 [ 0.198241] ACPI: Added _OSI(Module Device) [ 0.199012] ACPI: Added _OSI(Processor Device) [ 0.200011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.201023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206197] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.212573] ACPI: Interpreter enabled [ 0.215078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.216016] ACPI: Using IOAPIC for interrupt routing [ 0.218101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222356] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232467] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.234044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.241080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.246369] acpiphp: Slot [2] registered [ 0.247152] acpiphp: Slot [5] registered [ 0.249097] acpiphp: Slot [6] registered [ 0.250114] acpiphp: Slot [7] registered [ 0.251121] acpiphp: Slot [8] registered [ 0.253100] acpiphp: Slot [9] registered [ 0.254087] acpiphp: Slot [10] registered [ 0.255097] acpiphp: Slot [3] registered [ 0.256105] acpiphp: Slot [4] registered [ 0.258077] acpiphp: Slot [11] registered [ 0.259215] acpiphp: Slot [12] registered [ 0.260077] acpiphp: Slot [13] registered [ 0.261086] acpiphp: Slot [14] registered [ 0.263084] acpiphp: Slot [15] registered [ 0.264076] acpiphp: Slot [16] registered [ 0.265121] acpiphp: Slot [17] registered [ 0.268346] acpiphp: Slot [18] registered [ 0.270207] acpiphp: Slot [19] registered [ 0.272138] acpiphp: Slot [20] registered [ 0.274147] acpiphp: Slot [21] registered [ 0.275144] acpiphp: Slot [22] registered [ 0.277177] acpiphp: Slot [23] registered [ 0.279147] acpiphp: Slot [24] registered [ 0.280209] acpiphp: Slot [25] registered [ 0.282174] acpiphp: Slot [26] registered [ 0.284212] acpiphp: Slot [27] registered [ 0.286132] acpiphp: Slot [28] registered [ 0.287160] acpiphp: Slot [29] registered [ 0.289124] acpiphp: Slot [30] registered [ 0.291153] acpiphp: Slot [31] registered [ 0.292119] PCI host bridge to bus 0000:00 [ 0.293024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.295035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.297048] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.300030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.303027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.306028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.308185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.312000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.314323] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.325017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.330014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.333021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.335020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.339022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.342544] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.345899] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.348052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.351899] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.357026] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.370974] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.376035] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.382768] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.393018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.402017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.420018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.432187] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.439023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.445025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.467025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.478844] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.488019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.497025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.521025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.531789] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.539022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.544023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.562024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.569808] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.576020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.584020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.603028] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.613152] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.618020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.624025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.638029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.649080] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.651416] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.653392] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.655301] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.657232] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.662167] iommu: Default domain type: Passthrough [ 0.667378] SCSI subsystem initialized [ 0.669214] ACPI: bus type USB registered [ 0.670117] usbcore: registered new interface driver usbfs [ 0.672072] usbcore: registered new interface driver hub [ 0.673098] usbcore: registered new device driver usb [ 0.675177] pps_core: LinuxPPS API ver. 1 registered [ 0.677020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.680088] PTP clock support registered [ 0.683158] EDAC MC: Ver: 3.0.0 [ 0.685478] PCI: Using ACPI for IRQ routing [ 0.688132] NetLabel: Initializing [ 0.690012] NetLabel: domain hash size = 128 [ 0.691012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.693104] NetLabel: unlabeled traffic allowed by default [ 0.696229] vgaarb: loaded [ 0.697214] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.700020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.705019] clocksource: Switched to clocksource kvm-clock [ 0.820453] VFS: Disk quotas dquot_6.6.0 [ 0.822171] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.824508] *** VALIDATE ramfs *** [ 0.825941] *** VALIDATE hugetlbfs *** [ 0.827406] pnp: PnP ACPI init [ 0.829952] pnp: PnP ACPI: found 6 devices [ 0.853424] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.857650] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.860631] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.863328] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.866028] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.868684] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.872242] NET: Registered protocol family 2 [ 0.874992] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.880646] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.884683] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.891253] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.894853] TCP: Hash tables configured (established 65536 bind 65536) [ 0.897798] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.900910] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.903985] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.909216] NET: Registered protocol family 1 [ 0.912557] RPC: Registered named UNIX socket transport module. [ 0.914898] RPC: Registered udp transport module. [ 0.916735] RPC: Registered tcp transport module. [ 0.918408] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.920812] NET: Registered protocol family 44 [ 0.921981] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.923689] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.925863] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.928631] PCI: CLS 0 bytes, default 64 [ 0.929990] Unpacking initramfs... [ 2.378570] debug: unmapping init [mem 0xffff911a3cc54000-0xffff911a3ffbffff] [ 2.383112] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.385630] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.389257] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.891636] Initialise system trusted keyrings [ 2.893055] Key type blacklist registered [ 2.895186] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.903997] zbud: loaded [ 2.908585] *** VALIDATE nfs *** [ 2.909802] *** VALIDATE nfs4 *** [ 2.911372] pstore: using deflate compression [ 2.914783] Platform Keyring initialized [ 3.027892] NET: Registered protocol family 38 [ 3.030553] Key type asymmetric registered [ 3.032469] Asymmetric key parser 'x509' registered [ 3.034804] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.038492] io scheduler mq-deadline registered [ 3.039989] io scheduler kyber registered [ 3.041593] io scheduler bfq registered [ 3.043468] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.047474] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.050041] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.052834] ACPI: Power Button [PWRF] [ 3.058486] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.065194] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.078251] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.086940] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.105172] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.133396] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.160372] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.165034] Non-volatile memory driver v1.3 [ 3.166910] Linux agpgart interface v0.103 [ 3.198490] virtio_blk virtio1: [vda] 145800 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.203476] vda: detected capacity change from 0 to 74649600 [ 3.216871] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.220570] vdb: detected capacity change from 0 to 1073741824 [ 3.234536] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.238095] vdc: detected capacity change from 0 to 2621440000 [ 3.253139] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.256779] vdd: detected capacity change from 0 to 2621440000 [ 3.272915] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.276535] vde: detected capacity change from 0 to 4294967296 [ 3.292904] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.296546] vdf: detected capacity change from 0 to 4294967296 [ 3.303412] libphy: Fixed MDIO Bus: probed [ 3.316652] usbcore: registered new interface driver usbserial_generic [ 3.319230] usbserial: USB Serial support registered for generic [ 3.321330] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.325954] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.328395] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.331241] mousedev: PS/2 mouse device common for all mice [ 3.338588] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.339214] rtc_cmos 00:05: RTC can wake from S4 [ 3.343893] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.345119] rtc_cmos 00:05: registered as rtc0 [ 3.349229] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.349326] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.354603] intel_pstate: CPU model not supported [ 3.357739] hid: raw HID events driver (C) Jiri Kosina [ 3.359450] usbcore: registered new interface driver usbhid [ 3.361534] usbhid: USB HID core driver [ 3.363233] drop_monitor: Initializing network drop monitor service [ 3.365303] Initializing XFRM netlink socket [ 3.367344] NET: Registered protocol family 10 [ 3.370271] Segment Routing with IPv6 [ 3.371618] NET: Registered protocol family 17 [ 3.373555] mpls_gso: MPLS GSO support [ 3.379409] RAS: Correctable Errors collector initialized. [ 3.381095] AVX version of gcm_enc/dec engaged. [ 3.382494] AES CTR mode by8 optimization enabled [ 3.454175] sched_clock: Marking stable (3454144830, 0)->(4443525334, -989380504) [ 3.458028] registered taskstats version 1 [ 3.460213] Loading compiled-in X.509 certificates [ 3.462067] zswap: loaded using pool lzo/zbud [ 3.488453] Key type big_key registered [ 3.501413] Key type encrypted registered [ 3.503076] ima: No TPM chip found, activating TPM-bypass! [ 3.505381] ima: Allocated hash algorithm: sha1 [ 3.507139] ima: No architecture policies found [ 3.509047] evm: Initialising EVM extended attributes: [ 3.511090] evm: security.selinux [ 3.512288] evm: security.ima [ 3.513455] evm: security.capability [ 3.515025] evm: HMAC attrs: 0x1 [ 3.517647] rtc_cmos 00:05: setting system clock to 2026-07-26 14:56:08 UTC (1785077768) [ 3.524150] debug: unmapping init [mem 0xffffffff88003000-0xffffffff881fffff] [ 3.527564] debug: unmapping init [mem 0xffffffff86d82000-0xffffffff87058fff] [ 3.532082] Write protecting the kernel read-only data: 28672k [ 3.536031] debug: unmapping init [mem 0xffffffff85403000-0xffffffff855fffff] [ 3.539261] debug: unmapping init [mem 0xffffffff85d14000-0xffffffff85dfffff] [ 3.574994] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.583464] systemd[1]: Detected virtualization kvm. [ 3.585744] systemd[1]: Detected architecture x86-64. [ 3.588220] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.617808] systemd[1]: No hostname configured. [ 3.619678] systemd[1]: Set hostname to . [ 3.622047] random: systemd: uninitialized urandom read (16 bytes read) [ 3.624902] systemd[1]: Initializing machine ID from random generator. [ 3.753797] random: systemd: uninitialized urandom read (16 bytes read) [ 3.756611] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.760656] random: systemd: uninitialized urandom read (16 bytes read) [ 3.763769] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.767943] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.342299] device-mapper: uevent: version 1.0.3 [ 4.345079] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.094891] virtio_net virtio0 ens2: renamed from eth0 [ 5.110605] random: fast init done [ 5.153856] scsi host0: ata_piix [ 5.159732] scsi host1: ata_piix [ 5.162233] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.165782] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.681076] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.026711] random: crng init done [ 10.028107] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.535641] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.739141] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.007756] SELinux: Disabled at runtime. [ 12.068203] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.077845] systemd[1]: Detected virtualization kvm. [ 12.080116] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.583625] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.587440] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.592891] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.596438] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.598800] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.605194] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.612616] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ 12.742612] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.087637] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.475809] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.499722] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.621540] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.640787] EDAC sbridge: Ver: 1.1.2 [ 15.234396] Key type dns_resolver registered [ 15.563652] NFS: Registering the id_resolver key type [ 15.565845] Key type id_resolver registered [ 15.567691] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg352-server login: [ 25.794027] hrtimer: interrupt took 12799563 ns [ 59.645072] libcfs: loading out-of-tree module taints kernel. [ 59.698274] Key type ._llcrypt registered [ 59.700674] Key type .llcrypt registered [ 59.783324] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 76.946720] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 78.559375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 78.575456] alg: No test for adler32 (adler32-zlib) [ 80.102753] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 80.903034] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 82.703162] Key type lgssc registered [ 84.151394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 98.077829] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 129.701973] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 141.887681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 141.928290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.178418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.232689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.376865] Lustre: lustre-MDT0000: new disk, initializing [ 143.519423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.546418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 147.712435] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 152.595496] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 161.513948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 161.781741] Lustre: lustre-OST0000: new disk, initializing [ 161.787587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 161.794133] Lustre: Skipped 1 previous similar message [ 161.808533] Lustre: 6952:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 161.868480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.334927] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.552228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 168.566455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 168.605732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 182.751247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 182.890101] Lustre: lustre-OST0001: new disk, initializing [ 182.898733] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 182.905834] Lustre: 8017:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 182.962895] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 188.150357] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.459870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 189.472060] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 189.514512] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 199.396527] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 204.364840] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 210.223984] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing check_logdir /tmp/testlogs/ [ 215.285844] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing yml_node [ 219.974972] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 222.410920] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 224.775663] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 226.303780] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Jul 26 10:59:48 EDT 2026 [ 243.712830] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 245.028762] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 261.088360] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 261.090945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 261.100798] Lustre: Skipped 1 previous similar message [ 261.112710] Lustre: Skipped 1 previous similar message [ 266.208320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 266.212120] Lustre: Skipped 1 previous similar message [ 266.615258] Lustre: server umount lustre-MDT0000 complete [ 269.333675] LustreError: 5937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078034 with bad export cookie 15869628757578496522 [ 269.337818] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 269.347335] LustreError: 5937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 269.432443] Lustre: server umount lustre-OST0000 complete [ 272.336868] Lustre: server umount lustre-OST0001 complete [ 277.982304] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 285.632502] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 312.273839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 312.449322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 312.469078] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 312.547781] Lustre: lustre-MDT0000: new disk, initializing [ 312.600968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 312.636778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 317.672270] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 327.579226] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 333.457879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 333.661492] Lustre: lustre-OST0000: new disk, initializing [ 333.664874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 333.668071] Lustre: Skipped 1 previous similar message [ 333.672320] Lustre: 14105:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 333.728156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 335.665099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 335.669421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 335.694263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 339.716932] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 348.636688] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 353.086778] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 353.403976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 356.321296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.330635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 361.550926] Lustre: server umount lustre-OST0000 complete [ 365.564233] Lustre: server umount lustre-MDT0000 complete [ 369.966036] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 11:02:12 (1785078132) [ 372.206754] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 374.008992] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 11:02:16 (1785078136) [ 381.655852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 381.983916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.532595] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 394.071189] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 401.776822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 402.092480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.748910] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 420.130842] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 432.098493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 432.099286] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 432.106663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.908506] Lustre: server umount lustre-OST0000 complete [ 439.313890] Lustre: server umount lustre-MDT0000 complete [ 449.399338] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 451.842105] Key type lgssc unregistered [ 452.131903] LNet: 18462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.141573] LNetError: 18462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 452.166896] LNet: Removed LNI 192.168.203.152@tcp [ 453.035074] Key type .llcrypt unregistered [ 453.036853] Key type ._llcrypt unregistered [ 475.894190] Key type ._llcrypt registered [ 475.896468] Key type .llcrypt registered [ 476.063167] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 477.306323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 477.401047] alg: No test for adler32 (adler32-zlib) [ 478.456718] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 478.687414] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 480.423181] Key type lgssc registered [ 481.345255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 489.831841] loop: module loaded [ 489.922986] loop0: detected capacity change from 0 to 204800000 [ 490.163441] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 490.175652] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 490.625083] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 496.496926] loop0: detected capacity change from 0 to 204800000 [ 496.645190] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 496.658500] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 496.672028] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 502.086531] loop0: detected capacity change from 0 to 204800000 [ 507.349076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 507.389089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.661322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 508.716422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 508.824701] Lustre: lustre-MDT0000: new disk, initializing [ 508.883753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.908654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 513.553735] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 522.492886] loop1: detected capacity change from 0 to 204800000 [ 528.236384] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 528.587808] Lustre: lustre-OST07c6: new disk, initializing [ 528.597219] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 528.602825] Lustre: Skipped 1 previous similar message [ 528.608830] Lustre: 21646:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 528.752778] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 535.180486] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 538.213164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 538.226789] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 538.279727] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 540.775069] Lustre: 20634:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 540.792781] Lustre: 20634:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 540.798365] Lustre: 20634:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 540.808995] Lustre: 20634:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 540.822760] Lustre: 20634:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 540.830392] Lustre: 20634:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 555.366923] Lustre: 20635:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 555.375907] Lustre: 20635:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 555.382483] Lustre: 20635:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 555.390261] Lustre: 20635:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 555.403281] Lustre: 20635:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 555.414795] Lustre: 20635:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 560.251752] Lustre: 22310:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 560.265184] Lustre: 22310:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 560.272106] Lustre: 22310:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 560.282262] Lustre: 22310:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 560.290156] Lustre: 22310:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 560.303133] Lustre: 22310:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 560.308997] Lustre: 22310:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 560.316463] Lustre: 22310:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 560.323614] Lustre: 22310:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 560.332823] Lustre: 22310:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 560.336644] Lustre: 22310:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 560.342022] Lustre: 22310:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 563.554302] Lustre: 20633:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 563.561758] Lustre: 20633:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 563.567665] Lustre: 20633:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 563.574772] Lustre: 20633:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 563.580452] Lustre: 20633:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 563.589331] Lustre: 20633:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 563.600879] Lustre: 20633:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 563.610510] Lustre: 20633:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 563.615205] Lustre: 20633:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 563.623026] Lustre: 20633:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 563.628852] Lustre: 20633:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 563.635477] Lustre: 20633:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 568.191064] Lustre: 22310:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 568.209364] Lustre: 22310:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 568.219357] Lustre: 22310:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 568.226232] Lustre: 22310:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 568.232399] Lustre: 22310:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 568.237546] Lustre: 22310:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 568.246687] Lustre: 22310:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 568.261211] Lustre: 22310:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 568.276110] Lustre: 22310:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 568.288356] Lustre: 22310:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 568.299697] Lustre: 22310:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 568.306205] Lustre: 22310:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 579.488824] Lustre: 20634:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 579.497065] Lustre: 20634:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 579.504796] Lustre: 20634:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 579.510910] Lustre: 20634:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 579.519176] Lustre: 20634:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 579.526151] Lustre: 20634:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 579.532506] Lustre: 20634:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 579.542468] Lustre: 20634:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 579.551387] Lustre: 20634:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 579.557520] Lustre: 20634:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 579.566367] Lustre: 20634:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 579.576267] Lustre: 20634:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 630.241053] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 630.250160] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.268270] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 634.120961] Lustre: server umount lustre-OST07c6 complete [ 647.497957] Lustre: server umount lustre-MDT0000 complete [ 665.284537] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 667.896297] Key type lgssc unregistered [ 668.342844] LNet: 24204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.353699] LNetError: 24204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.379031] LNet: Removed LNI 192.168.203.152@tcp [ 669.261062] Key type .llcrypt unregistered [ 669.266423] Key type ._llcrypt unregistered [ 684.748710] Key type ._llcrypt registered [ 684.750509] Key type .llcrypt registered [ 684.843969] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 685.602320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 685.701489] alg: No test for adler32 (adler32-zlib) [ 686.727962] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 686.885836] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 688.561341] Key type lgssc registered [ 689.731613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 696.154739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 696.180042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.554893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 700.882621] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 707.392182] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 713.176500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 713.578348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 719.389189] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 722.798098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:65) [ 726.180897] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 743.408039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 743.413497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 743.438332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 744.553223] Lustre: server umount lustre-OST0000 complete [ 748.671492] Lustre: server umount lustre-MDT0000 complete [ 758.736715] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 762.966232] Key type lgssc unregistered [ 763.576950] LNet: 28074:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.587872] LNetError: 28074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.618802] LNet: Removed LNI 192.168.203.152@tcp [ 764.792511] Key type .llcrypt unregistered [ 764.794227] Key type ._llcrypt unregistered [ 788.216654] Key type ._llcrypt registered [ 788.218039] Key type .llcrypt registered [ 788.296161] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 789.042955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 789.073522] alg: No test for adler32 (adler32-zlib) [ 790.115372] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 790.241667] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 791.847182] Key type lgssc registered [ 793.005514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 800.764567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 800.786724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.237446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 806.380558] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 813.168216] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 819.398216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 819.689385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 825.059259] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 827.911188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 833.164559] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 843.238333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 843.258099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 843.265979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 848.353820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 849.695775] Lustre: server umount lustre-OST0000 complete [ 854.246714] Lustre: server umount lustre-MDT0000 complete [ 864.598663] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 867.498960] Key type lgssc unregistered [ 867.766627] LNet: 31767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.778354] LNetError: 31767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.792672] LNet: Removed LNI 192.168.203.152@tcp [ 868.623392] Key type .llcrypt unregistered [ 868.627987] Key type ._llcrypt unregistered [ 891.662837] Key type ._llcrypt registered [ 891.667426] Key type .llcrypt registered [ 891.808524] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 892.741613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 892.782766] alg: No test for adler32 (adler32-zlib) [ 893.840880] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 894.065649] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 895.743284] Key type lgssc registered [ 897.057777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 905.554304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 905.572509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 907.034511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 912.754530] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 923.494804] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 931.182941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 931.756070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.804917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 938.199578] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 946.659590] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 953.313736] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.327762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 954.274628] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 955.122703] Lustre: server umount lustre-MDT0000 complete [ 967.471486] LustreError: 33165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.52@tcp arrived at 1785078732 with bad export cookie 17031256985726173831 [ 969.588937] LustreError: 33167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785078734 with bad export cookie 17031256985726173250 [ 969.596074] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 969.599458] LustreError: 33167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 975.842821] Lustre: server umount lustre-OST0000 complete [ 989.399390] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 992.433401] Key type lgssc unregistered [ 992.762594] LNet: 35782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.770243] LNetError: 35782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.783865] LNet: Removed LNI 192.168.203.152@tcp [ 993.669431] Key type .llcrypt unregistered [ 993.678344] Key type ._llcrypt unregistered [ 1015.724390] Key type ._llcrypt registered [ 1015.732357] Key type .llcrypt registered [ 1015.868497] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1016.881595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1016.907701] alg: No test for adler32 (adler32-zlib) [ 1017.954385] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1018.246501] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1019.991161] Key type lgssc registered [ 1021.404964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1031.464714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1031.501540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1033.603641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.884362] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1051.969473] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1058.729668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1059.267456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1065.811203] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1073.139697] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1078.656615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 1085.895065] Lustre: lustre-OST0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 1088.992432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1088.998935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.005748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1091.006201] Lustre: server umount lustre-OST0000 complete [ 1114.693905] Lustre: server umount lustre-MDT0000 complete [ 1125.794738] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1128.706686] Key type lgssc unregistered [ 1129.017367] LNet: 39794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.047707] LNetError: 39794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.077585] LNet: Removed LNI 192.168.203.152@tcp [ 1130.034059] Key type .llcrypt unregistered [ 1130.040086] Key type ._llcrypt unregistered [ 1153.525814] Key type ._llcrypt registered [ 1153.527806] Key type .llcrypt registered [ 1153.657664] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1155.385168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1155.403215] alg: No test for adler32 (adler32-zlib) [ 1156.663726] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1157.038994] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1158.740117] Key type lgssc registered [ 1160.315382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1169.727285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1169.774250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1171.411116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.342725] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1184.292946] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1190.958698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1191.338650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.480189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 1196.615359] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1206.564284] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1213.858223] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1218.579265] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1230.604375] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-2611h16m32s [ 1232.171751] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1235.363155] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 1236.421716] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 1237.472412] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.501059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1240.454076] Lustre: server umount lustre-MDT0000 complete [ 1243.961647] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1250.319244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.396025] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1250.701150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1250.769740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:225) [ 1255.564607] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1259.779944] Lustre: DEBUG MARKER: Wait for df (31366) ... [ 1261.635820] Lustre: DEBUG MARKER: done [ 1265.137549] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1270.240850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.254078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1270.261888] Lustre: Skipped 1 previous similar message [ 1273.140357] Lustre: server umount lustre-OST0000 complete [ 1277.305746] Lustre: server umount lustre-MDT0000 complete [ 1289.761649] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1293.004481] Key type lgssc unregistered [ 1293.451857] LNet: 45038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.461354] LNetError: 45038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.480749] LNet: Removed LNI 192.168.203.152@tcp [ 1294.448856] Key type .llcrypt unregistered [ 1294.451545] Key type ._llcrypt unregistered [ 1338.200702] Key type ._llcrypt registered [ 1338.202467] Key type .llcrypt registered [ 1338.277519] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1339.226513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1339.260938] alg: No test for adler32 (adler32-zlib) [ 1340.436924] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1340.697472] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1342.383356] Key type lgssc registered [ 1343.581899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1351.023118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1351.062612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.367324] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1352.386831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1352.564520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1357.090528] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1364.195768] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1370.190688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1370.373607] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1370.563168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.551994] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1378.927052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:257) [ 1386.648127] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1394.004289] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-2611h19m16s [ 1395.563965] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1401.534348] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1404.316943] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1404.390839] LustreError: 47053:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1424.870369] Lustre: lustre-MDT0000: Client 3896a195-711f-4ed2-a9e6-3c0c0796ef00 (at 192.168.203.52@tcp) reconnecting [ 1444.431157] LustreError: 47053:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1446.107261] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1447.866373] Lustre: DEBUG MARKER: done [ 1456.104502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1456.118805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.154910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1458.664781] Lustre: server umount lustre-OST0000 complete [ 1462.190902] Lustre: server umount lustre-MDT0000 complete [ 1471.300992] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1474.308358] Key type lgssc unregistered [ 1474.673248] LNet: 50248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.680550] LNetError: 50248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.751039] LNet: Removed LNI 192.168.203.152@tcp [ 1476.697329] Key type .llcrypt unregistered [ 1476.699253] Key type ._llcrypt unregistered [ 1514.177788] Key type ._llcrypt registered [ 1514.182396] Key type .llcrypt registered [ 1514.338766] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1515.145228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1515.163052] alg: No test for adler32 (adler32-zlib) [ 1516.314599] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1516.543327] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1518.224701] Key type lgssc registered [ 1519.472872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1523.015923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1523.057793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1524.396748] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1524.430336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1524.676753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1525.720794] Lustre: Failing over lustre-MDT0000 [ 1526.129508] Lustre: server umount lustre-MDT0000 complete [ 1532.594278] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:21:35 (1785079295) [ 1538.729751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.035763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1542.956157] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1549.329450] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1555.606206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1555.870970] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1556.052401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.853666] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1563.175190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:259 to 0x240000400:289) [ 1569.075554] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1574.974451] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1583.145855] Lustre: server umount lustre-MDT0000 complete [ 1586.548161] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1590.792194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1598.173984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.301462] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1598.558450] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.609476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.705310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1599.583175] Lustre: 51747:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785079348/real 1785079348] req@00000000d43bdc99 x1871791292171392/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785079364 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1602.080974] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1604.072513] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1605.087859] Lustre: 51749:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785079353/real 1785079353] req@0000000046ae123f x1871791292171648/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785079369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1608.853546] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1610.207161] Lustre: 51748:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785079358/real 1785079358] req@00000000af667272 x1871791292171904/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785079374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1614.306683] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1614.327902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1616.283505] Lustre: server umount lustre-MDT0000 complete [ 1621.686505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1629.415675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.562940] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1629.886957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.951781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1633.849646] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1634.993871] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1640.290648] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1648.214941] Lustre: server umount lustre-MDT0000 complete [ 1653.668130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1656.611667] Lustre: DEBUG MARKER: files compared the same [ 1658.227339] LustreError: 52815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785079423 with bad export cookie 10888488672767727384 [ 1658.237951] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1668.720955] Lustre: server umount lustre-OST0000 complete [ 1681.840190] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1684.505546] Key type lgssc unregistered [ 1684.760160] LNet: 59057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.770531] LNetError: 59057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.795570] LNet: Removed LNI 192.168.203.152@tcp [ 1685.663664] Key type .llcrypt unregistered [ 1685.668123] Key type ._llcrypt unregistered [ 1704.376216] Key type ._llcrypt registered [ 1704.378034] Key type .llcrypt registered [ 1704.528706] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1705.751945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1705.802414] alg: No test for adler32 (adler32-zlib) [ 1706.795245] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1706.990793] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1708.639178] Key type lgssc registered [ 1709.747531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1717.546329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1717.557777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.032031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.743614] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug +malloc all [ 1732.037599] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1739.099929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1739.356455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.466120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:35 to 0x240000401:65) [ 1744.813434] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug +malloc all [ 1752.404284] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1762.267169] Lustre: server umount lustre-OST0000 complete [ 1775.224810] Lustre: server umount lustre-MDT0000 complete [ 1785.009316] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1787.410706] Key type lgssc unregistered [ 1787.639138] LNet: 62871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.647738] LNetError: 62871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.670342] LNet: Removed LNI 192.168.203.152@tcp [ 1788.318653] Key type .llcrypt unregistered [ 1788.320669] Key type ._llcrypt unregistered [ 1808.355867] Key type ._llcrypt registered [ 1808.357827] Key type .llcrypt registered [ 1808.478507] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1809.622807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1809.641072] alg: No test for adler32 (adler32-zlib) [ 1810.742689] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1810.987645] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1812.655183] Key type lgssc registered [ 1813.765303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1820.419558] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1820.436210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1835.935410] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1841.055837] LustreError: 64240:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 1867.679589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.460059] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1882.358562] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1889.923611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1889.981824] Lustre: *** cfs_fail_loc=706, val=0*** [ 1890.182826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.223687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:35 to 0x240000401:97) [ 1894.039781] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1900.069025] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1905.632453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1905.647943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.662806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1907.838907] Lustre: server umount lustre-OST0000 complete [ 1912.055171] Lustre: server umount lustre-MDT0000 complete [ 1928.596133] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1931.387977] Key type lgssc unregistered [ 1931.614632] LNet: 66744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.623720] LNetError: 66744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.632665] LNet: Removed LNI 192.168.203.152@tcp [ 1932.291309] Key type .llcrypt unregistered [ 1932.293799] Key type ._llcrypt unregistered [ 1955.689931] Key type ._llcrypt registered [ 1955.691693] Key type .llcrypt registered [ 1955.785712] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1956.747102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1956.885140] alg: No test for adler32 (adler32-zlib) [ 1957.905408] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 1958.090803] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1959.735535] Key type lgssc registered [ 1960.735995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1968.804531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1968.844866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.696427] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1982.924810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1983.326195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.513334] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1996.862304] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1996.873091] Lustre: MGS: Client 240f8cbd-4756-4ebd-8d92-fbfcf6619a4c (at 0@lo) reconnecting [ 1997.237341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.302953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:35 to 0x240000401:129) [ 2001.295067] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2017.760891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2017.776265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.807496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2018.792391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2021.614873] Lustre: server umount lustre-OST0000 complete [ 2025.827914] Lustre: server umount lustre-MDT0000 complete [ 2029.580040] Lustre: server umount MGS complete [ 2037.227318] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:29:59 (1785079799) [ 2059.338665] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 2068.631928] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2098.431481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.356194] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2111.595116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2111.812382] Lustre: lustre-OST0000: new disk, initializing [ 2111.818991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2111.832420] Lustre: 74379:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2111.922368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.575881] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2126.021589] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2132.277513] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2132.290418] Lustre: MGS: Client bc82e737-941f-45fd-8eba-fa4499da15f1 (at 0@lo) reconnecting [ 2132.429091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2132.439760] LustreError: 73659:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2132.465325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2132.533157] Lustre: lustre-MDT0000: new disk, initializing [ 2132.665639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.695664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2132.805156] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2133.857950] Lustre: 75445:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000003625469b x1871791755645056/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785079914 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2137.061521] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2142.963459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2142.976383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2143.030742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2163.686030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2163.694210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.706218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2167.461296] Lustre: server umount lustre-OST0000 complete [ 2185.184291] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2185.376074] Lustre: server umount lustre-MDT0000 complete [ 2189.091288] Lustre: server umount MGS complete [ 2197.096252] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:32:39 (1785079959) [ 2210.472457] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2212.960808] Key type lgssc unregistered [ 2213.223696] LNet: 77436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2213.238049] LNetError: 77436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.310208] LNet: Removed LNI 192.168.203.152@tcp [ 2215.052822] Key type .llcrypt unregistered [ 2215.054981] Key type ._llcrypt unregistered [ 2231.777317] Key type ._llcrypt registered [ 2231.780786] Key type .llcrypt registered [ 2231.894965] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2232.797869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2232.827995] alg: No test for adler32 (adler32-zlib) [ 2233.930487] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2234.249304] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2235.911180] Key type lgssc registered [ 2236.991628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2243.917956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2243.937227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.515618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.644947] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2256.583880] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2262.536514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2262.968586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.955809] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2270.121593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2275.965978] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2284.764259] Lustre: server umount lustre-OST0000 complete [ 2299.078052] Lustre: server umount lustre-MDT0000 complete [ 2310.221685] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2313.202689] Key type lgssc unregistered [ 2313.466879] LNet: 81078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2313.472513] LNetError: 81078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2313.508600] LNet: Removed LNI 192.168.203.152@tcp [ 2314.397155] Key type .llcrypt unregistered [ 2314.399891] Key type ._llcrypt unregistered [ 2329.289854] Key type ._llcrypt registered [ 2329.293852] Key type .llcrypt registered [ 2329.415973] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2330.598033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2330.804403] alg: No test for adler32 (adler32-zlib) [ 2331.988363] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2332.281150] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2333.951183] Key type lgssc registered [ 2335.280161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2341.287967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2341.294865] LustreError: 82144:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2341.411197] LustreError: 82160:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2341.420989] LustreError: 82144:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4883 [ 2341.433749] LustreError: 82144:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2341.442969] LustreError: 82144:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2341.452187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2343.049223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.192919] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing lsmod [ 2349.811358] LustreError: 82651:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2349.834182] LustreError: 82654:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2349.839254] LustreError: 82651:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4980 [ 2349.842777] LustreError: 82651:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2349.856953] LustreError: 82651:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2349.859469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2350.102989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2352.178231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 2353.061104] Lustre: server umount lustre-MDT0000 complete [ 2356.450548] LustreError: 82175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785080121 with bad export cookie 9841673101950463754 [ 2356.467851] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2367.045404] Lustre: server umount lustre-OST0000 complete [ 2374.358968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.774476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2378.540590] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2385.415424] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2391.450634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2391.744274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.173763] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2399.919599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:97) [ 2403.692501] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2410.469056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2410.476138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.493562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.593487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.784742] Lustre: server umount lustre-OST0000 complete [ 2419.460214] Lustre: server umount lustre-MDT0000 complete [ 2427.967495] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2430.370664] Key type lgssc unregistered [ 2430.626944] LNet: 85931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2430.635337] LNetError: 85931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2430.665149] LNet: Removed LNI 192.168.203.152@tcp [ 2431.387894] Key type .llcrypt unregistered [ 2431.390035] Key type ._llcrypt unregistered [ 2450.819864] Key type ._llcrypt registered [ 2450.822763] Key type .llcrypt registered [ 2450.921072] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2451.681804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2451.754636] alg: No test for adler32 (adler32-zlib) [ 2452.825568] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2453.072654] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2454.791124] Key type lgssc registered [ 2455.818279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2462.612858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2462.633547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.952342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.117655] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2472.795914] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2478.285348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2478.536350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.572209] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2488.823422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 2490.120975] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2495.636516] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2498.815445] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2502.354256] Lustre: server umount lustre-OST0000 complete [ 2516.064235] Lustre: server umount lustre-MDT0000 complete [ 2527.401447] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2529.905641] Key type lgssc unregistered [ 2530.130892] LNet: 89849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2530.139367] LNetError: 89849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2530.162964] LNet: Removed LNI 192.168.203.152@tcp [ 2530.945458] Key type .llcrypt unregistered [ 2530.947146] Key type ._llcrypt unregistered [ 2545.078652] Key type ._llcrypt registered [ 2545.081436] Key type .llcrypt registered [ 2545.165808] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2546.181696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2546.221974] alg: No test for adler32 (adler32-zlib) [ 2547.234772] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2547.474223] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2549.128040] Key type lgssc registered [ 2550.572871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2557.813854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2557.836057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.253485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.514515] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2570.994388] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2577.577262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2578.126802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.278390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 2584.861956] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2592.570581] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2605.026824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.035626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2605.051734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2607.582220] Lustre: server umount lustre-OST0000 complete [ 2611.735341] Lustre: server umount lustre-MDT0000 complete [ 2619.972556] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2622.364256] Key type lgssc unregistered [ 2622.666825] LNet: 93471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2622.674938] LNetError: 93471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2622.687619] LNet: Removed LNI 192.168.203.152@tcp [ 2623.574092] Key type .llcrypt unregistered [ 2623.579039] Key type ._llcrypt unregistered [ 2642.474305] Key type ._llcrypt registered [ 2642.476366] Key type .llcrypt registered [ 2642.588817] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2643.185438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2643.220140] alg: No test for adler32 (adler32-zlib) [ 2644.202416] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2644.458110] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2646.127175] Key type lgssc registered [ 2647.060942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2653.018759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2653.046844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.404738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.914486] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2664.727854] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2670.387825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.692590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2675.803886] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2679.913635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 2682.009617] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2688.723164] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2692.028216] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2692.038960] Lustre: Skipped 1 previous similar message [ 2695.954646] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2695.958946] Lustre: Skipped 1 previous similar message [ 2697.786406] Lustre: lustre-MDT0000: nosquash_nids is cleared [ 2697.816360] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2703.450702] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.52@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2703.462981] Lustre: Skipped 1 previous similar message [ 2712.109061] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.203.52@tcp 192.168.0.[2,10]@tcp [ 2715.696399] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.52/24@tcp in log lustre-MDT0000 [ 2715.703668] Lustre: Skipped 1 previous similar message [ 2718.732940] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.52/24@tcp [ 2728.942151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2728.942257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.956171] LustreError: Skipped 2 previous similar messages [ 2729.000879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2731.493118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2733.211849] Lustre: server umount lustre-OST0000 complete [ 2737.609237] Lustre: server umount lustre-MDT0000 complete [ 2747.446499] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2749.720263] Key type lgssc unregistered [ 2749.952442] LNet: 98666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2749.957615] LNetError: 98666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2749.976050] LNet: Removed LNI 192.168.203.152@tcp [ 2750.660938] Key type .llcrypt unregistered [ 2750.663178] Key type ._llcrypt unregistered [ 2777.778368] Key type ._llcrypt registered [ 2777.780985] Key type .llcrypt registered [ 2777.890254] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2779.035293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2779.128554] alg: No test for adler32 (adler32-zlib) [ 2780.151886] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2780.380883] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2782.063520] Key type lgssc registered [ 2783.116488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2790.281839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2790.295520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.650737] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2791.669827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.672868] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.52/24@tcp [ 2795.477475] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2802.043937] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2807.769111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2808.055711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2812.917500] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2816.252579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:201 to 0x240000400:225) [ 2819.029435] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2831.842785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2831.845746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.862938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2834.918277] Lustre: server umount lustre-OST0000 complete [ 2839.755735] Lustre: server umount lustre-MDT0000 complete [ 2851.555677] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2854.328774] Key type lgssc unregistered [ 2854.523242] LNet: 102702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2854.527378] LNetError: 102702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2854.539983] LNet: Removed LNI 192.168.203.152@tcp [ 2855.386656] Key type .llcrypt unregistered [ 2855.388831] Key type ._llcrypt unregistered [ 2876.100794] Key type ._llcrypt registered [ 2876.107278] Key type .llcrypt registered [ 2876.241975] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 2891.078896] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2892.449808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2892.459629] alg: No test for adler32 (adler32-zlib) [ 2893.501457] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 2893.777193] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2895.447184] Key type lgssc registered [ 2896.584313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2920.685825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2920.729036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.110212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2922.184438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2922.280796] Lustre: lustre-MDT0000: new disk, initializing [ 2922.408619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.428947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2926.743601] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2935.489217] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2941.665597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.992363] Lustre: lustre-OST0000: new disk, initializing [ 2941.995574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2942.000046] Lustre: Skipped 1 previous similar message [ 2942.013278] Lustre: 106684:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2942.093567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2948.524147] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2949.764688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2949.775840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2949.807261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2957.895817] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2962.728540] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2962.958546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2964.964418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.987284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2970.084167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2970.873830] Lustre: server umount lustre-OST0000 complete [ 2974.581608] Lustre: server umount lustre-MDT0000 complete [ 2982.756924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.240125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.970463] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2993.020368] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2998.960400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2999.234458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3004.765898] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3012.012339] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3016.435323] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3016.614238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3023.943113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3024.150330] Lustre: lustre-OST0001: new disk, initializing [ 3024.157547] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3024.173703] Lustre: 110621:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3024.273646] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3029.158405] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3030.641801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3030.653308] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3030.685932] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3036.482675] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3036.701513] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3041.114904] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8af643087800.ost_server_uuid 50 [ 3042.458352] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8af643087800.ost_server_uuid in FULL state after 0 sec [ 3048.855299] Lustre: server umount lustre-OST0001 complete [ 3050.982359] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3050.986852] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3053.776891] Lustre: server umount lustre-OST0000 complete [ 3067.024937] Lustre: server umount lustre-MDT0000 complete [ 3082.649558] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 3085.820749] Key type lgssc unregistered [ 3086.142984] LNet: 112992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3086.160915] LNetError: 112992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3086.193091] LNet: Removed LNI 192.168.203.152@tcp [ 3087.251871] Key type .llcrypt unregistered [ 3087.256438] Key type ._llcrypt unregistered [ 3120.830599] Key type ._llcrypt registered [ 3120.834063] Key type .llcrypt registered [ 3120.928152] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 3134.476869] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3135.916281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3135.948170] alg: No test for adler32 (adler32-zlib) [ 3137.055924] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 3137.332081] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 3138.999168] Key type lgssc registered [ 3140.212708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3165.449518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3165.464297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.733639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3166.788427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3167.000518] Lustre: lustre-MDT0000: new disk, initializing [ 3167.075668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.094802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3171.612644] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3180.912107] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3187.373863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3187.733421] Lustre: lustre-OST0000: new disk, initializing [ 3187.741357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3187.749596] Lustre: Skipped 1 previous similar message [ 3187.756780] Lustre: 117533:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3187.838384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3193.122087] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3197.001311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3197.008863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3197.032924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3201.654341] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3206.256596] Lustre: Failing over lustre-OST0000 [ 3206.413622] Lustre: server umount lustre-OST0000 complete [ 3223.519277] Lustre: 115021:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785080972/real 1785080972] req@0000000048100bad x1871792991908096/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785080988 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3223.519955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.561599] Lustre: 115021:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3224.132984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3224.504584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.517463] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3225.010924] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3225.635048] Lustre: 115021:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785080974/real 1785080974] req@00000000e3760e60 x1871792991908608/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1785080990 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3228.687549] Lustre: 115019:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785080977/real 1785080977] req@00000000b1c4f230 x1871792991908992/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785080993 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3230.040708] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3233.613862] Lustre: Failing over lustre-MDT0000 [ 3233.628456] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3233.653637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3233.956599] Lustre: server umount lustre-MDT0000 complete [ 3254.240110] Lustre: 115020:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785081003/real 1785081003] req@000000002358d37b x1871792991914624/t0(0) o400->MGC192.168.203.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785081019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3254.270569] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3255.404672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.718894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3264.777882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.828359] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3268.662544] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3270.143146] Lustre: lustre-OST0000: Recovery over after 0:45, of 2 clients 2 recovered and 0 were evicted. [ 3270.147555] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3273.198942] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3276.296633] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3276.357599] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3276.395765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3281.888334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3281.896227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.908334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3283.432177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3284.024706] Lustre: server umount lustre-OST0000 complete [ 3288.346569] Lustre: server umount lustre-MDT0000 complete [ 3300.275804] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 3303.202298] Key type lgssc unregistered [ 3303.530868] LNet: 120992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3303.537772] LNetError: 120992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3304.617716] LNet: Removed LNI 192.168.203.152@tcp [ 3305.428481] Key type .llcrypt unregistered [ 3305.432054] Key type ._llcrypt unregistered [ 3339.166376] Key type ._llcrypt registered [ 3339.169769] Key type .llcrypt registered [ 3339.258537] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3340.438867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3340.478832] alg: No test for adler32 (adler32-zlib) [ 3341.451952] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 3341.692318] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 3343.359221] Key type lgssc registered [ 3344.376357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3351.331829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3351.348179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.813499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3357.438239] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3366.305829] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3370.478608] Lustre: server umount lustre-MDT0000 complete [ 3383.296429] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 3392.475831] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3421.199242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.516687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3421.545643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3421.641411] Lustre: lustre-MDT0000: new disk, initializing [ 3421.765680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.794240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3426.410660] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3435.383272] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3442.278415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3442.601907] Lustre: lustre-OST0000: new disk, initializing [ 3442.607454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3442.613861] Lustre: Skipped 1 previous similar message [ 3442.625694] Lustre: 126424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3442.692545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3448.923807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3448.934186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3448.963777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3449.404797] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3458.634878] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3462.871770] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3464.443706] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:53:46 (1785081226) [ 3694.564534] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.603936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3699.680441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3704.801668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3708.383145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3708.620334] Lustre: server umount lustre-MDT0000 complete [ 3712.509084] LustreError: 126423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785081477 with bad export cookie 2950026731928536345 [ 3712.521679] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3718.789201] Lustre: server umount lustre-OST0000 complete [ 3728.561699] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 3737.537094] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3765.749104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3765.924391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3765.951773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3766.002483] Lustre: lustre-MDT0000: new disk, initializing [ 3766.052816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3766.066588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3770.174845] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3778.878549] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3785.892813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3786.323317] Lustre: lustre-OST0000: new disk, initializing [ 3786.326121] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3786.331681] Lustre: Skipped 1 previous similar message [ 3786.336558] Lustre: 131607:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3786.424390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3792.289459] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3793.007231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3793.031015] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3793.070703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3800.884873] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3804.998210] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3805.401481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3808.228197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3808.237662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.249858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3813.346947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3814.174672] Lustre: server umount lustre-OST0000 complete [ 3818.470309] Lustre: server umount lustre-MDT0000 complete [ 3827.919666] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:59:50 (1785081590) [ 3841.926236] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 3850.796913] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3878.438473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3878.640021] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3878.698561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3878.823433] Lustre: lustre-MDT0000: new disk, initializing [ 3878.895442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3878.915264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3882.964538] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3891.098072] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3897.704783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3898.053255] Lustre: lustre-OST0000: new disk, initializing [ 3898.062530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3898.066871] Lustre: Skipped 1 previous similar message [ 3898.071917] Lustre: 136488:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3898.173589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3904.465595] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3906.125505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3906.138733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3906.187745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3913.618433] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3921.383794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3921.401753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3921.417307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3926.201486] Lustre: server umount lustre-OST0000 complete [ 3930.432797] Lustre: server umount lustre-MDT0000 complete [ 3938.052535] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:01:40 (1785081700) [ 3949.632695] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 3959.366289] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3990.088788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3990.372442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3990.396886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3990.529415] Lustre: lustre-MDT0000: new disk, initializing [ 3990.637663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3990.652559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3994.840835] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4003.585784] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4009.956772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4010.264448] Lustre: 141187:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4017.021325] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4020.355145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4020.391895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4026.107363] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4035.561499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4035.572788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4040.462295] Lustre: server umount lustre-OST0000 complete [ 4044.089231] Lustre: server umount lustre-MDT0000 complete [ 4055.769109] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 4059.097647] Key type lgssc unregistered [ 4059.422386] LNet: 142835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4059.431451] LNetError: 142835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4059.449653] LNet: Removed LNI 192.168.203.152@tcp [ 4060.356423] Key type .llcrypt unregistered [ 4060.360137] Key type ._llcrypt unregistered [ 4083.098636] Key type ._llcrypt registered [ 4083.102462] Key type .llcrypt registered [ 4083.218828] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 4084.320948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4084.352342] alg: No test for adler32 (adler32-zlib) [ 4085.444581] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 4085.730324] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 4087.391269] Key type lgssc registered [ 4088.546589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4095.378632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4095.402744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4096.770188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4101.013732] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4107.266967] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4112.420440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4112.745120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.847714] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4121.970123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 4127.587937] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4147.681444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4147.707156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4151.110508] Lustre: server umount lustre-OST0000 complete [ 4155.225795] Lustre: server umount lustre-MDT0000 complete [ 4167.892194] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 4170.138595] Key type lgssc unregistered [ 4170.542946] LNet: 146695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4170.559456] LNetError: 146695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4171.621791] LNet: Removed LNI 192.168.203.152@tcp [ 4172.937110] Key type .llcrypt unregistered [ 4172.944481] Key type ._llcrypt unregistered [ 4195.456895] Key type ._llcrypt registered [ 4195.461521] Key type .llcrypt registered [ 4195.567558] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 4196.719847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4196.742296] alg: No test for adler32 (adler32-zlib) [ 4197.800914] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 4198.027039] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 4199.743196] Key type lgssc registered [ 4201.038355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4208.383470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4208.416983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4209.812331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4213.866561] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4220.722597] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4226.405920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4226.698455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4233.693788] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4234.859427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:65) [ 4241.001927] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4248.987113] Lustre: lustre-OST0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 4250.595223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4250.607724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4250.625234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4253.759658] Lustre: server umount lustre-OST0000 complete [ 4260.080040] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8af643edf000.ost_server_uuid 50 [ 4261.726615] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4281.767674] Lustre: server umount lustre-MDT0000 complete [ 4289.640848] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:07:31 (1785082051) [ 4295.780279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4296.325520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4300.907586] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4307.939463] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4314.399519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4314.834835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4320.663351] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4322.040740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 4328.448325] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4335.251237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4335.474451] Lustre: lustre-OST0001: new disk, initializing [ 4335.485611] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4335.495265] Lustre: 152843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4335.634534] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4342.687336] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4343.470909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 4343.479868] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 4343.536025] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 4352.289693] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4358.054081] Lustre: lustre-OST0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 4358.118527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4358.131376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.141637] Lustre: lustre-OST0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 4363.155164] Lustre: Skipped 2 previous similar messages [ 4368.261370] Lustre: lustre-OST0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 4368.285883] Lustre: Skipped 1 previous similar message [ 4369.375494] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4369.549965] Lustre: server umount lustre-OST0000 complete [ 4376.048474] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4376.266083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4378.499664] LustreError: 153644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.528662] LustreError: 153644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4379.630691] LustreError: 151830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.633363] LustreError: 151830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.867809] LustreError: 151830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.884820] LustreError: 151830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4394.979567] LustreError: 153644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.981465] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4395.019746] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4395.046574] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4395.054728] Lustre: Skipped 1 previous similar message [ 4396.119256] Lustre: server umount lustre-OST0001 complete [ 4400.113529] Lustre: server umount lustre-MDT0000 complete [ 4420.504962] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:09:42 (1785082182) [ 4427.333527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.648951] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4427.703622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4427.859818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4431.958068] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4437.753314] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4444.190455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4444.322024] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4444.441085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4450.997766] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4452.935376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 4459.689611] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4466.393949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4466.526332] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4473.098818] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4480.502483] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4483.554559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4483.558596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4483.570856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4490.050359] Lustre: server umount lustre-OST0000 complete [ 4491.700063] LustreError: 156539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4509.666233] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4509.675175] LustreError: 156539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4509.676804] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4509.721417] LustreError: 156539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4511.932384] Lustre: server umount lustre-OST0001 complete [ 4515.750245] Lustre: server umount lustre-MDT0000 complete [ 4534.859838] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:11:37 (1785082297) [ 4548.169547] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 4557.263777] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 4583.130365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4583.328132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4583.351530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4583.461700] Lustre: lustre-MDT0000: new disk, initializing [ 4583.574445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4583.581847] Lustre: Skipped 1 previous similar message [ 4583.610107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4588.637438] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4597.624752] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4604.445969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4604.807877] Lustre: lustre-OST0000: new disk, initializing [ 4604.810745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4604.813355] Lustre: Skipped 1 previous similar message [ 4604.818419] Lustre: 162944:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4610.188668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4610.204376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4610.254462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4611.604398] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4621.292755] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4625.870576] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4626.054853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4630.513251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4630.525019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4630.529461] Lustre: Skipped 4 previous similar messages [ 4642.272118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4642.562526] Lustre: server umount lustre-OST0000 complete [ 4646.985849] Lustre: server umount lustre-MDT0000 complete [ 4655.243421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4656.033064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4656.043194] Lustre: Skipped 1 previous similar message [ 4661.013416] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4667.668259] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4674.974717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4682.177262] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4689.341134] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4693.860913] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4694.123570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4696.036288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4696.041726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4696.065968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4696.077121] Lustre: Skipped 2 previous similar messages [ 4710.367296] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4710.556163] Lustre: server umount lustre-OST0000 complete [ 4718.705583] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4749.060196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4755.784115] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4761.699348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4761.721144] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4762.344477] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4766.806771] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 4772.321131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4772.339448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4776.500269] Lustre: server umount lustre-OST0000 complete [ 4780.518678] Lustre: server umount lustre-MDT0000 complete [ 4788.187683] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:15:50 (1785082550) [ 4793.945222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4794.467598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4794.473771] Lustre: Skipped 2 previous similar messages [ 4799.112945] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4806.035609] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4812.617862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4819.005728] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4825.731878] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4830.091396] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4830.308152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4836.934834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4837.074741] Lustre: lustre-OST0001: new disk, initializing [ 4837.078495] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4837.085130] Lustre: 171218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4843.716330] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4847.222718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 4847.235810] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 4847.262525] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 4852.394228] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4856.904235] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4857.062623] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4862.443573] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4862.458034] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4862.463854] Lustre: Skipped 3 previous similar messages [ 4873.183119] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4873.260665] Lustre: server umount lustre-OST0001 complete [ 4877.797925] LustreError: 169937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4879.441724] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4879.687962] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4907.467395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4912.580479] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4912.609549] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4912.647049] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4919.553522] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4920.752529] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4922.850578] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4928.710256] Lustre: server umount lustre-OST0001 complete [ 4943.338504] LustreError: 172669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4943.383971] LustreError: 172669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 4952.847695] Lustre: server umount lustre-OST0000 complete [ 4963.445529] Lustre: server umount lustre-MDT0000 complete [ 4983.460227] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:19:05 (1785082745) [ 4989.699172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4989.945581] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4989.978305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4994.953927] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5001.949194] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5007.818290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5008.011474] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5014.373869] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5021.144427] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5029.314398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5029.358676] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5036.092597] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5043.581331] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5045.396303] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5050.503355] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5050.762572] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5056.471811] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8af64305e000.ost_server_uuid 50 [ 5058.312931] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8af64305e000.ost_server_uuid in IDLE state after 0 sec [ 5059.332461] Lustre: Permanently deactivating lustre-OST0001 [ 5059.362535] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5063.587455] Lustre: Permanently reactivating lustre-OST0001 [ 5063.604518] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5063.609320] Lustre: Skipped 1 previous similar message [ 5068.261529] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5068.274700] LustreError: Skipped 1 previous similar message [ 5068.286912] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5068.320663] Lustre: Skipped 2 previous similar messages [ 5074.307657] Lustre: server umount lustre-OST0001 complete [ 5077.999070] LustreError: 176441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5078.030846] LustreError: 176441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5093.346340] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5093.554219] Lustre: server umount lustre-OST0000 complete [ 5097.882237] Lustre: server umount lustre-MDT0000 complete [ 5117.931358] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:21:20 (1785082880) [ 5125.584552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5126.051974] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5126.087279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5126.095871] Lustre: Skipped 1 previous similar message [ 5126.248041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5126.256579] Lustre: Skipped 6 previous similar messages [ 5130.671943] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5138.327517] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5145.374173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5145.605192] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5145.637665] Lustre: Permanently deactivating lustre-OST0000 [ 5145.648199] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5152.316386] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5153.835589] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5153.848181] LustreError: 181990:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 5153.849484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 5159.619721] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5166.698752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5166.774847] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5172.318147] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:65) [ 5173.114580] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5180.121369] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5181.726813] Lustre: Permanently reactivating lustre-OST0000 [ 5181.747242] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5181.756947] Lustre: Skipped 1 previous similar message [ 5188.707938] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5188.714863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.730876] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5188.732645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 5197.807573] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5197.811787] Lustre: Skipped 13 previous similar messages [ 5197.986940] Lustre: server umount lustre-OST0001 complete [ 5207.653920] Lustre: server umount lustre-OST0000 complete [ 5211.191162] Lustre: server umount lustre-MDT0000 complete [ 5222.619710] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5225.625218] Key type lgssc unregistered [ 5225.926739] LNet: 184342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5225.942733] LNetError: 184342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5225.963720] LNet: Removed LNI 192.168.203.152@tcp [ 5226.948899] Key type .llcrypt unregistered [ 5226.954400] Key type ._llcrypt unregistered [ 5253.413991] Key type ._llcrypt registered [ 5253.415781] Key type .llcrypt registered [ 5253.600222] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 5268.924339] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5270.501519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5270.547514] alg: No test for adler32 (adler32-zlib) [ 5271.599965] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 5271.899837] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5273.583217] Key type lgssc registered [ 5274.536874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5299.843720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5299.864407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.122385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5301.148575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5301.259283] Lustre: lustre-MDT0000: new disk, initializing [ 5301.376175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5301.403721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5305.857138] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5314.604293] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5320.859474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5321.100340] Lustre: lustre-OST0000: new disk, initializing [ 5321.104204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5321.108485] Lustre: Skipped 1 previous similar message [ 5321.113581] Lustre: 188377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5321.186570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5327.133448] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5327.573196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5327.585086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5327.628663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5336.030350] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5340.258177] LustreError: 187387:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5347.605522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5347.788703] Lustre: lustre-OST0001: new disk, initializing [ 5347.791663] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5347.798803] Lustre: 189486:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5347.877943] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5350.304169] LustreError: 187387:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5350.341097] LustreError: 187385:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5352.971947] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5356.185339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 5356.190819] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 5356.218609] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 5360.431202] LustreError: 187385:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5361.294548] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5362.817043] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5364.632196] Lustre: lustre-OST0001: Not available for connect from 192.168.203.52@tcp (stopping) [ 5366.755426] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5366.760510] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.782680] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5369.738146] Lustre: lustre-OST0001: Not available for connect from 192.168.203.52@tcp (stopping) [ 5370.482467] Lustre: server umount lustre-OST0001 complete [ 5376.996330] LustreError: 188361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.020994] LustreError: 188361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5382.116473] LustreError: 190150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5387.235020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5387.236333] LustreError: 188364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5387.270093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5392.353824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5392.354764] LustreError: 188364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5392.682462] Lustre: server umount lustre-OST0000 complete [ 5403.040523] Lustre: server umount lustre-MDT0000 complete [ 5414.194879] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5417.375510] Key type lgssc unregistered [ 5417.735281] LNet: 191314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5417.748689] LNetError: 191314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5417.771358] LNet: Removed LNI 192.168.203.152@tcp [ 5418.616663] Key type .llcrypt unregistered [ 5418.619275] Key type ._llcrypt unregistered [ 5454.765887] Key type ._llcrypt registered [ 5454.767368] Key type .llcrypt registered [ 5454.864680] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5457.199141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5457.229578] alg: No test for adler32 (adler32-zlib) [ 5458.558317] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 5458.997202] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5460.703194] Key type lgssc registered [ 5461.859971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5469.771123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5469.806644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5471.204920] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5471.243267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5471.424596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5475.989452] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5482.441669] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5488.467283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5488.753342] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5488.956653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5495.081283] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5498.892763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4 to 0x240000400:33) [ 5502.044684] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5555.170686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5555.185753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5555.206756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5557.678719] Lustre: server umount lustre-OST0000 complete [ 5561.412858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5571.423349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5571.925876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5576.038184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.064839] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5577.592632] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5584.405877] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5627.365502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5627.378239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.400881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5629.265852] Lustre: server umount lustre-OST0000 complete [ 5633.200264] Lustre: server umount lustre-MDT0000 complete [ 5643.304056] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5645.747978] Key type lgssc unregistered [ 5646.035571] LNet: 197613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5646.042879] LNetError: 197613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5646.061275] LNet: Removed LNI 192.168.203.152@tcp [ 5646.760869] Key type .llcrypt unregistered [ 5646.765419] Key type ._llcrypt unregistered [ 5667.907683] Key type ._llcrypt registered [ 5667.910386] Key type .llcrypt registered [ 5668.037924] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5669.115108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5669.141902] alg: No test for adler32 (adler32-zlib) [ 5670.184561] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 5670.355332] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5671.991184] Key type lgssc registered [ 5673.106242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5680.555979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5680.593786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.222609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5686.127586] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5694.124288] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5702.842802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5703.239757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5707.378461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:42 to 0x240000400:65) [ 5709.526859] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5717.198605] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5743.587359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5743.591147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.620749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5746.045843] Lustre: server umount lustre-OST0000 complete [ 5750.281308] Lustre: server umount lustre-MDT0000 complete [ 5760.939409] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5763.983242] Key type lgssc unregistered [ 5764.322490] LNet: 202219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5764.330357] LNetError: 202219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5765.349720] LNet: Removed LNI 192.168.203.152@tcp [ 5766.245888] Key type .llcrypt unregistered [ 5766.248725] Key type ._llcrypt unregistered [ 5783.353535] Key type ._llcrypt registered [ 5783.359154] Key type .llcrypt registered [ 5783.508087] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5784.431271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5784.507489] alg: No test for adler32 (adler32-zlib) [ 5785.667035] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 5786.069262] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5787.783263] Key type lgssc registered [ 5788.769762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5796.169607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5796.186975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.619633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5802.924108] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5811.009308] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5818.027136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5818.538493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5822.640404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 5824.711414] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5833.086448] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5848.547255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5848.548799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.604996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5851.440531] Lustre: server umount lustre-OST0000 complete [ 5855.286802] Lustre: server umount lustre-MDT0000 complete [ 5866.031795] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5868.480462] Key type lgssc unregistered [ 5868.778090] LNet: 206003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5868.786492] LNetError: 206003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5868.803303] LNet: Removed LNI 192.168.203.152@tcp [ 5869.810789] Key type .llcrypt unregistered [ 5869.815347] Key type ._llcrypt unregistered [ 5894.318657] Key type ._llcrypt registered [ 5894.325847] Key type .llcrypt registered [ 5894.492193] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5896.513186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5896.559070] alg: No test for adler32 (adler32-zlib) [ 5897.625885] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 5897.938040] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5899.695166] Key type lgssc registered [ 5901.102035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5909.400828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5909.423237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5910.774306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5914.508704] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5920.729236] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5927.211132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5927.530136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5935.534625] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5935.739433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 5943.632701] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5966.820755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5966.828513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.850323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5970.235620] Lustre: server umount lustre-OST0000 complete [ 5973.868849] Lustre: server umount lustre-MDT0000 complete [ 5984.164098] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5986.816870] Key type lgssc unregistered [ 5987.119789] LNet: 210610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5987.130865] LNetError: 210610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5987.143258] LNet: Removed LNI 192.168.203.152@tcp [ 5988.020713] Key type .llcrypt unregistered [ 5988.024574] Key type ._llcrypt unregistered [ 6002.177454] Key type ._llcrypt registered [ 6002.180712] Key type .llcrypt registered [ 6002.272109] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6003.097773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6003.111051] alg: No test for adler32 (adler32-zlib) [ 6004.229952] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6004.528931] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6006.239171] Key type lgssc registered [ 6007.387664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6014.707365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6014.731503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6016.238549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6020.514429] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6026.634716] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6033.384077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6033.732504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6039.395448] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6041.986498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 6046.737946] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6057.440956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6057.450966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6057.466910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6062.562387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6062.748646] Lustre: server umount lustre-OST0000 complete [ 6066.812488] Lustre: server umount lustre-MDT0000 complete [ 6077.109809] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6079.671176] Key type lgssc unregistered [ 6079.982873] LNet: 214415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6079.995377] LNetError: 214415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6080.037633] LNet: Removed LNI 192.168.203.152@tcp [ 6080.853829] Key type .llcrypt unregistered [ 6080.859618] Key type ._llcrypt unregistered [ 6091.430293] Key type ._llcrypt registered [ 6091.433366] Key type .llcrypt registered [ 6091.518892] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6105.520979] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 6118.409343] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6119.707662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6119.731674] alg: No test for adler32 (adler32-zlib) [ 6120.762581] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6121.037163] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6122.743262] Key type lgssc registered [ 6124.053958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6153.306855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6153.342128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.600992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6154.632168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6154.695193] Lustre: lustre-MDT0000: new disk, initializing [ 6154.808892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6154.832429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6158.635360] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6167.719883] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6174.081587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6174.272535] Lustre: lustre-OST0000: new disk, initializing [ 6174.275548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6174.278747] Lustre: Skipped 1 previous similar message [ 6174.282756] Lustre: 218532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6174.336388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.013384] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6183.505503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6183.514332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6183.547517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6190.869649] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6195.988162] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6196.211118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6198.757642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6198.758710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6198.795364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6203.875134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6204.475423] Lustre: server umount lustre-OST0000 complete [ 6208.476458] Lustre: server umount lustre-MDT0000 complete [ 6217.273450] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:39:39 (1785083979) [ 6223.607715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6224.271740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6228.513799] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6234.755831] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6241.031021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.430462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6247.691589] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6254.970464] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6265.312236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6265.322683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.334326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6270.432995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6270.706836] Lustre: server umount lustre-OST0000 complete [ 6274.850967] Lustre: server umount lustre-MDT0000 complete [ 6285.752943] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6288.724759] Key type lgssc unregistered [ 6289.230112] LNet: 222891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6289.253272] LNetError: 222891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6290.277805] LNet: Removed LNI 192.168.203.152@tcp [ 6291.301764] Key type .llcrypt unregistered [ 6291.308032] Key type ._llcrypt unregistered [ 6316.226209] Key type ._llcrypt registered [ 6316.229821] Key type .llcrypt registered [ 6316.333240] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 6330.899961] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6332.241979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6332.309308] alg: No test for adler32 (adler32-zlib) [ 6333.451306] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6333.656644] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6335.319737] Key type lgssc registered [ 6336.480364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6371.048222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6371.078102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6372.428227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6372.485220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6372.567741] Lustre: lustre-MDT0000: new disk, initializing [ 6372.650704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6372.675346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6376.535873] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6384.926630] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6391.118278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6391.326217] Lustre: lustre-OST2710: new disk, initializing [ 6391.332850] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6391.341488] Lustre: Skipped 1 previous similar message [ 6391.346154] Lustre: 227301:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6391.417539] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6396.492827] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6400.123175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:2710:ost [ 6400.133692] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:2710:ost] [ 6400.161333] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x240000400 [ 6405.242811] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6411.796727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6411.944749] Lustre: lustre-OST03e8: new disk, initializing [ 6411.949267] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6411.957214] Lustre: 228360:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6412.039975] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6417.657822] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6419.102394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:3e8:ost [ 6419.109831] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:3e8:ost] [ 6419.142924] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x280000400 [ 6423.989838] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6430.563973] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6430.807936] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6435.949707] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6436.208270] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6442.847524] Lustre: server umount lustre-MDT0000 complete [ 6447.007198] LustreError: 228359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785084211 with bad export cookie 17850948279432805690 [ 6447.017728] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6456.306992] Lustre: server umount lustre-OST2710 complete [ 6471.031352] Lustre: server umount lustre-OST03e8 complete [ 6485.818720] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 6495.571316] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6523.271476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6523.585944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6523.651678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6523.764324] Lustre: lustre-MDT0000: new disk, initializing [ 6523.860791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6523.892233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6528.307700] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6536.949345] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6543.192969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6543.476335] Lustre: lustre-OST0000: new disk, initializing [ 6543.479559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6543.482973] Lustre: Skipped 1 previous similar message [ 6543.490771] Lustre: 233450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6543.587065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6549.151117] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6553.704770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6553.711890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6553.749360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6557.910601] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6563.319231] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6563.678877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6568.928590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6568.939401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6568.959807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6574.050792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6579.168886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6580.191502] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6580.396690] Lustre: server umount lustre-OST0000 complete [ 6585.057821] Lustre: server umount lustre-MDT0000 complete [ 6595.988817] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:45:57 (1785084357) [ 6597.976758] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6599.988286] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:46:02 (1785084362) [ 6604.128165] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6627.692816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.960440] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6627.996297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6628.215432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6632.793727] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6653.874149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6654.136057] Lustre: Denying initial registration attempt from nid 192.168.203.152@tcp, specified as failover [ 6654.143548] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6654.150036] LustreError: 237358:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 6654.160754] LustreError: 237358:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6654.165304] LustreError: 237358:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6654.281266] Lustre: server umount lustre-OST0000 complete [ 6654.283583] LustreError: 237358:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6660.454911] Lustre: server umount lustre-MDT0000 complete [ 6670.697676] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6673.478474] Key type lgssc unregistered [ 6673.764032] LNet: 238277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6673.778701] LNetError: 238277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6673.818835] LNet: Removed LNI 192.168.203.152@tcp [ 6674.678139] Key type .llcrypt unregistered [ 6674.679829] Key type ._llcrypt unregistered [ 6684.526601] Key type ._llcrypt registered [ 6684.528933] Key type .llcrypt registered [ 6684.658777] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6685.386058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6685.524402] alg: No test for adler32 (adler32-zlib) [ 6686.580593] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6686.822953] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6688.514105] Key type lgssc registered [ 6689.860178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6731.299293] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6738.735190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6738.749502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6739.989790] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6740.009534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6740.209992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6743.722836] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6748.825513] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6767.920526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6768.161576] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6768.401747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6773.890873] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6778.700556] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6782.434551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6782.453575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6782.463967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6786.051651] Lustre: server umount lustre-OST0000 complete [ 6789.429204] Lustre: server umount lustre-MDT0000 complete [ 6798.187073] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6800.511221] Key type lgssc unregistered [ 6800.760651] LNet: 243281:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6800.770758] LNetError: 243281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6800.792446] LNet: Removed LNI 192.168.203.152@tcp [ 6801.516274] Key type .llcrypt unregistered [ 6801.518319] Key type ._llcrypt unregistered [ 6821.427111] Key type ._llcrypt registered [ 6821.436618] Key type .llcrypt registered [ 6821.529941] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6822.637579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6822.766336] alg: No test for adler32 (adler32-zlib) [ 6823.890228] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6824.138569] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6825.847172] Key type lgssc registered [ 6827.053134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6834.635945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6834.670902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.116949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6840.283982] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6846.380747] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6853.950693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6854.313818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6860.038621] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6866.678952] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6877.155777] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6877.183988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6878.625494] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 6880.748963] Lustre: server umount lustre-MDT0000 complete [ 6886.583576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6893.753338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6893.847123] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6894.252903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6894.304094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:166 to 0x240000400:193) [ 6897.318945] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6897.368750] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6903.176661] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6904.413372] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6909.922679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6909.927595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6909.946337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6912.995092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6918.114708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6923.231193] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6923.400708] Lustre: server umount lustre-OST0000 complete [ 6927.154281] Lustre: server umount lustre-MDT0000 complete [ 6937.590910] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6940.414168] Key type lgssc unregistered [ 6940.788488] LNet: 248670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6940.799367] LNetError: 248670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6940.811355] LNet: Removed LNI 192.168.203.152@tcp [ 6941.603614] Key type .llcrypt unregistered [ 6941.605762] Key type ._llcrypt unregistered [ 6963.251935] Key type ._llcrypt registered [ 6963.255170] Key type .llcrypt registered [ 6963.370408] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6964.125285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6964.238381] alg: No test for adler32 (adler32-zlib) [ 6965.236951] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 6965.461896] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6967.136734] Key type lgssc registered [ 6968.248188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6975.113776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6975.135133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.466733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6979.976454] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6986.525744] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6990.233954] Lustre: server umount lustre-MDT0000 complete [ 7003.433807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7003.720319] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7003.773496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7003.937262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7007.961860] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7015.194297] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7024.556308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7024.754938] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7024.761036] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7024.771771] LustreError: 252528:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 7024.777193] LustreError: 252528:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7024.783110] LustreError: 252528:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7024.858614] Lustre: server umount lustre-OST0000 complete [ 7024.860228] LustreError: 252528:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7034.980972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7035.187676] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7035.341309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7041.079990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:166 to 0x240000400:225) [ 7042.072898] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7049.173648] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7056.902631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7057.009985] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7057.063514] Lustre: lustre-OST0001: new disk, initializing [ 7057.070843] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7057.077978] Lustre: 254062:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7057.173086] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7062.650879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 7062.662956] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 7062.696964] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 7063.918473] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7071.132045] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7077.857717] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.876078] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7080.405674] Lustre: server umount lustre-OST0001 complete [ 7087.073822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7087.080762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7087.093176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7088.096714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7088.105840] LustreError: 252995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.128085] LustreError: 252995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7093.218047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7093.235942] LustreError: 252995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7098.337363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7098.348240] LustreError: 252996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7099.359233] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7099.612955] Lustre: server umount lustre-OST0000 complete [ 7103.895130] Lustre: server umount lustre-MDT0000 complete [ 7115.196543] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 7118.330601] Key type lgssc unregistered [ 7118.753807] LNet: 255815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7118.765363] LNetError: 255815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7118.787669] LNet: Removed LNI 192.168.203.152@tcp [ 7119.773508] Key type .llcrypt unregistered [ 7119.777516] Key type ._llcrypt unregistered [ 7168.773592] Key type ._llcrypt registered [ 7168.775689] Key type .llcrypt registered [ 7168.881262] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 7180.958858] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7181.847772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7181.865993] alg: No test for adler32 (adler32-zlib) [ 7182.994254] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 7183.153970] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 7184.831267] Key type lgssc registered [ 7185.998441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7214.123369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7214.148407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7215.450312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7215.489160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7215.580256] Lustre: lustre-MDT0000: new disk, initializing [ 7215.688119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7215.708086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7219.714351] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7228.229715] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7234.216965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7234.413133] Lustre: lustre-OST0000: new disk, initializing [ 7234.415354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7234.418917] Lustre: Skipped 1 previous similar message [ 7234.424745] Lustre: 260751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7234.491715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7239.808878] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7241.770433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7241.774658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7241.793783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7248.351348] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7252.391285] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7252.588477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7257.056613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7257.070841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7257.092719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7260.740762] Lustre: server umount lustre-OST0000 complete [ 7263.836246] Lustre: server umount lustre-MDT0000 complete [ 7269.999302] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:57:12 (1785085032) [ 7276.431814] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:57:18 (1785085038) [ 7284.957830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7285.660850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7290.328791] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7297.928081] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7305.172994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7305.725344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7313.311416] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7322.204865] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7330.233530] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7331.812379] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7333.596336] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7335.337897] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7337.279238] Lustre: Failing over lustre-MDT0000 [ 7337.763563] Lustre: server umount lustre-MDT0000 complete [ 7356.630640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7356.780364] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7356.901946] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7357.318608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.378970] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7358.368796] Lustre: 258237:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785085106/real 1785085106] req@000000005fcdb3c7 x1871797234461696/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785085122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7362.029518] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7362.080759] Lustre: 258238:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785085111/real 1785085111] req@00000000badd5a0a x1871797234461952/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785085127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7362.434539] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 7362.509383] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 7362.540908] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7362.595839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:65) [ 7367.648847] Lustre: 258236:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785085116/real 1785085116] req@0000000006197542 x1871797234462208/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785085132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7371.363303] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7373.221900] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7376.755892] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7378.485040] Lustre: Failing over lustre-MDT0000 [ 7378.783401] Lustre: server umount lustre-MDT0000 complete [ 7397.370418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7397.535966] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7397.799400] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7397.867077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7397.937917] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7398.392875] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 7398.440295] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 7398.498330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:97) [ 7398.944070] Lustre: 258238:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785085147/real 1785085147] req@0000000009647b07 x1871797234471680/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785085163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7401.935618] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7402.990527] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7404.959151] Lustre: 258237:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785085153/real 1785085153] req@0000000049616ce2 x1871797234471936/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785085169 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7411.148846] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7412.892316] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7418.339579] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7418.352974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7423.457067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7424.687447] Lustre: server umount lustre-MDT0000 complete [ 7428.078978] LustreError: 263011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085193 with bad export cookie 5323932594237111159 [ 7428.085323] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7428.094344] LustreError: 263011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7428.238930] Lustre: server umount lustre-OST0000 complete [ 7441.351975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7441.722262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7445.548104] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7452.652054] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7460.348131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7466.863671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:129) [ 7467.505354] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7475.559884] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7481.695619] Lustre: server umount lustre-OST0000 complete [ 7495.829333] Lustre: server umount lustre-MDT0000 complete [ 7507.148956] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 7509.901874] Key type lgssc unregistered [ 7510.293633] LNet: 270611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7510.303814] LNetError: 270611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7510.320082] LNet: Removed LNI 192.168.203.152@tcp [ 7511.327661] Key type .llcrypt unregistered [ 7511.330868] Key type ._llcrypt unregistered [ 7538.144169] Key type ._llcrypt registered [ 7538.146407] Key type .llcrypt registered [ 7538.246848] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7539.230565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7539.290492] alg: No test for adler32 (adler32-zlib) [ 7540.381857] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 7540.640037] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 7542.303195] Key type lgssc registered [ 7543.672286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7551.701285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7551.720500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7553.245582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7557.531730] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7563.914889] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7571.276784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7571.727591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7578.191432] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7578.936161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:161) [ 7585.682967] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7598.824042] Lustre: server umount lustre-MDT0000 complete [ 7602.908056] LustreError: 272105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085367 with bad export cookie 8540025265051430429 [ 7602.921305] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7612.606417] Lustre: server umount lustre-OST0000 complete [ 7630.120268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7630.662517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7637.111696] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7644.598849] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7651.351499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7651.759673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7655.862992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 7657.908535] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7666.329518] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7681.505948] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7681.547589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7683.838429] Lustre: server umount lustre-MDT0000 complete [ 7687.837402] LustreError: 274708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785085452 with bad export cookie 8540025265051431220 [ 7687.842652] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7687.854402] LustreError: 274708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7688.014245] Lustre: server umount lustre-OST0000 complete [ 7700.609640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7701.110787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7705.560465] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7713.973839] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7721.616805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7722.043480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7726.125052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:225) [ 7727.610600] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7735.309192] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7741.310576] Lustre: server umount lustre-OST0000 complete [ 7753.305810] Lustre: server umount lustre-MDT0000 complete [ 7763.654347] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 7766.423900] Key type lgssc unregistered [ 7766.757139] LNet: 279846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7766.771569] LNetError: 279846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7766.802208] LNet: Removed LNI 192.168.203.152@tcp [ 7767.564709] Key type .llcrypt unregistered [ 7767.567201] Key type ._llcrypt unregistered [ 7791.453783] Key type ._llcrypt registered [ 7791.458494] Key type .llcrypt registered [ 7791.606617] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7792.603367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7792.682050] alg: No test for adler32 (adler32-zlib) [ 7793.764170] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 7794.001492] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 7795.671223] Key type lgssc registered [ 7796.921709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7804.861494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7804.896206] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7804.904331] LustreError: 281284:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7804.956760] LustreError: 281284:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7804.967616] LustreError: 281284:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7804.975241] LustreError: 281284:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7804.987130] LustreError: 281284:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7811.979450] LDISKFS-fs (dm-1): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7811.992731] LustreError: 281681:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7812.023577] LustreError: 281681:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7812.030538] LustreError: 281681:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7812.038800] LustreError: 281681:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7812.047071] LustreError: 281681:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7829.526421] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 7832.111320] Key type lgssc unregistered [ 7832.411609] LNet: 282532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7832.420726] LNetError: 282532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7832.442906] LNet: Removed LNI 192.168.203.152@tcp [ 7833.066507] Key type .llcrypt unregistered [ 7833.070380] Key type ._llcrypt unregistered [ 7849.545967] Key type ._llcrypt registered [ 7849.556314] Key type .llcrypt registered [ 7849.721768] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 7865.327720] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7866.723766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7866.769898] alg: No test for adler32 (adler32-zlib) [ 7867.930944] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 7868.193623] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 7869.895155] Key type lgssc registered [ 7871.183363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7901.497766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7901.532464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7902.981213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7903.018635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7903.154592] Lustre: lustre-MDT0000: new disk, initializing [ 7903.286301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7903.317873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7908.296478] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7918.636984] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7926.088375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7926.360910] Lustre: lustre-OST0000: new disk, initializing [ 7926.364760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7926.370634] Lustre: Skipped 1 previous similar message [ 7926.379456] Lustre: 286275:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7926.476607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7933.533328] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7936.074323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7936.082207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7936.114072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7944.175239] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7949.743192] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7950.075514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7956.459806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7956.467675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7956.503178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7958.330248] Lustre: server umount lustre-OST0000 complete [ 7962.156531] Lustre: server umount lustre-MDT0000 complete [ 7970.152083] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7971.823210] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:08:54 (1785085734) [ 7978.006273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7978.546952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7983.121412] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7990.066874] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7996.785477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7997.120192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8003.122530] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8009.749265] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8015.885489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8016.085520] Lustre: lustre-OST0001: new disk, initializing [ 8016.088992] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8016.105975] Lustre: 290187:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8016.196625] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8021.704216] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8025.237814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 8025.244024] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 8025.278534] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 8031.063470] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8035.302501] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8035.330723] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8037.817358] Lustre: lustre-OST0001: Not available for connect from 192.168.203.52@tcp (stopping) [ 8040.419120] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8040.652892] Lustre: server umount lustre-OST0001 complete [ 8050.656853] LustreError: 289178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.686449] LustreError: 289178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8055.789087] LustreError: 289176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8059.879516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8059.890488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8059.904123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8060.908324] LustreError: 289176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8063.898396] Lustre: server umount lustre-OST0000 complete [ 8074.022988] Lustre: server umount lustre-MDT0000 complete [ 8084.781569] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 8087.356222] Key type lgssc unregistered [ 8087.658334] LNet: 291965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8087.662443] LNetError: 291965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8087.678124] LNet: Removed LNI 192.168.203.152@tcp [ 8088.483114] Key type .llcrypt unregistered [ 8088.484741] Key type ._llcrypt unregistered [ 8132.536400] Key type ._llcrypt registered [ 8132.539177] Key type .llcrypt registered [ 8132.662713] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 8133.418925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8133.459685] alg: No test for adler32 (adler32-zlib) [ 8134.405828] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 8134.586107] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 8136.223340] Key type lgssc registered [ 8137.397796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8146.716396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8146.753127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8148.089933] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8148.121793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8148.317187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8153.942703] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8162.087928] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8164.504566] Lustre: server umount lustre-MDT0000 complete [ 8174.297893] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:12:16 (1785085936) [ 8180.774697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8181.234344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8186.450465] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8194.252320] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8201.617388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8201.885520] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8202.113450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8209.555482] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8218.344054] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8227.052884] Lustre: Permanently deactivating lustre-OST0000 [ 8227.084146] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8234.561956] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8237.179215] LustreError: 297875:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8237.188860] LustreError: 297875:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8237.193882] LustreError: 297875:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8239.621231] LustreError: 297923:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8239.626341] LustreError: 297923:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8247.845384] Lustre: server umount lustre-OST0000 complete [ 8251.553421] LustreError: 298174:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8251.564239] LustreError: 298174:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8263.761705] Lustre: server umount lustre-MDT0000 complete [ 8273.420571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.376354] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8281.736648] LustreError: 299132:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8288.769702] Lustre: 299472:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.152@tcp [ 8290.974720] Lustre: server umount MGS complete [ 8299.575894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8300.125355] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8300.181344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8305.311976] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8312.497607] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8314.419353] Lustre: Permanently reactivating lustre-OST0000 [ 8314.433733] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8314.446937] Lustre: Skipped 1 previous similar message [ 8330.896391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8331.476217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8338.484599] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8346.805426] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8353.256042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8353.271693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.297529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8358.374704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8359.291402] Lustre: server umount lustre-OST0000 complete [ 8363.826447] Lustre: server umount lustre-MDT0000 complete [ 8375.379912] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 8378.571887] Key type lgssc unregistered [ 8378.921706] LNet: 302811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8378.933201] LNetError: 302811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8378.944064] LNet: Removed LNI 192.168.203.152@tcp [ 8379.829704] Key type .llcrypt unregistered [ 8379.836587] Key type ._llcrypt unregistered [ 8396.640556] Key type ._llcrypt registered [ 8396.644057] Key type .llcrypt registered [ 8396.751475] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 8413.091866] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 8413.911382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8413.931975] alg: No test for adler32 (adler32-zlib) [ 8415.098769] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 8415.338737] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 8417.039170] Key type lgssc registered [ 8418.883277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8449.570742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8449.648181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8450.985698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8451.033255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8451.121374] Lustre: lustre-MDT0000: new disk, initializing [ 8451.225454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8451.240057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8455.859967] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8465.413707] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8471.705526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8471.919892] Lustre: lustre-OST0000: new disk, initializing [ 8471.923115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8471.926642] Lustre: Skipped 1 previous similar message [ 8471.934235] Lustre: 306550:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8472.007116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8478.361575] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8479.837227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8479.844755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8479.895874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8487.544258] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8492.282429] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8492.550434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8495.085833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8495.090253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8495.113370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8500.192866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8500.650803] Lustre: server umount lustre-OST0000 complete [ 8505.156412] Lustre: server umount lustre-MDT0000 complete [ 8515.359592] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8517.403234] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:17:59 (1785086279) [ 8524.269603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8524.639660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8527.940271] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8534.680721] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8541.825611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8542.255708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8548.967458] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8557.464579] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8559.164301] Lustre: ctl-lustre-MDT0000: [0x0000000280000400-0x00000002c0000400]:0:mdt sequences allocated: rc = 0 [ 8562.863758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:mdt [ 8569.833178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8569.836134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8569.868121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8573.602346] Lustre: server umount lustre-OST0000 complete [ 8577.484382] Lustre: server umount lustre-MDT0000 complete [ 8588.756602] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 8591.701816] Key type lgssc unregistered [ 8591.943712] LNet: 311146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8591.957443] LNetError: 311146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8591.974787] LNet: Removed LNI 192.168.203.152@tcp [ 8592.987400] Key type .llcrypt unregistered [ 8592.992515] Key type ._llcrypt unregistered [ 8640.783626] Key type ._llcrypt registered [ 8640.790543] Key type .llcrypt registered [ 8640.963932] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 8641.212374] Key type .llcrypt unregistered [ 8641.214286] Key type ._llcrypt unregistered [ 8655.405735] Key type ._llcrypt registered [ 8655.408112] Key type .llcrypt registered [ 8655.495404] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 8656.212494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8656.234715] alg: No test for adler32 (adler32-zlib) [ 8657.358459] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 8657.668348] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 8659.359173] Key type lgssc registered [ 8660.311154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8676.481296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8676.545599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8677.759750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8677.795264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8677.849551] Lustre: lustre-MDT0000: new disk, initializing [ 8677.902326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8677.918872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8682.311937] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8692.170860] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8700.958646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8701.317902] Lustre: lustre-OST0000: new disk, initializing [ 8701.326734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8701.332597] Lustre: Skipped 1 previous similar message [ 8701.360740] Lustre: 315618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8701.462835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8707.812856] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8709.304549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8709.319559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8709.390429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8716.100427] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8724.449257] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8724.462346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8728.469139] Lustre: server umount lustre-MDT0000 complete [ 8732.144172] LustreError: 314609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785086497 with bad export cookie 3305027026158518716 [ 8732.145608] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8732.162629] LustreError: 314609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8732.488932] Lustre: server umount lustre-OST0000 complete [ 8742.818181] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:21:45 (1785086505) [ 8750.252110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.006979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8755.324993] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8761.733384] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8781.234067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8781.428634] Lustre: Found index 0 for lustre-OST0000, updating log [ 8781.559209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8788.246461] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 8795.333958] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8796.908967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5 to 0x240000400:33) [ 8807.393026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8807.404065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8807.430341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8808.489521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8810.831294] Lustre: server umount lustre-OST0000 complete [ 8825.998312] Lustre: server umount lustre-MDT0000 complete [ 8833.571442] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:23:16 (1785086596) [ 8844.063609] LNet: 1 local NIs in recovery (showing 1): 192.168.203.152@tcp [ 9450.208116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9451.112476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9456.379255] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9464.042602] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9483.561414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9483.734883] Lustre: Found index 0 for lustre-OST0000, updating log [ 9483.978662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9489.589924] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9496.680880] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9498.820635] Lustre: server umount lustre-OST0000 complete [ 9502.558652] Lustre: server umount lustre-MDT0000 complete [ 9510.690368] LNet: 1 local NIs in recovery (showing 1): 192.168.203.152@tcp [ 9530.218231] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 9542.273712] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 9574.475889] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:35:36 (1785087336) [ 9588.520469] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 9591.353780] Key type lgssc unregistered [ 9591.657829] LNet: 327666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9591.662813] LNetError: 327666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9713.518304] LNet: Removed LNI 192.168.203.152@tcp [ 9714.613350] Key type .llcrypt unregistered [ 9714.615965] Key type ._llcrypt unregistered [ 9730.454571] Key type ._llcrypt registered [ 9730.457239] Key type .llcrypt registered [ 9730.551974] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 9731.531551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9731.646145] alg: No test for adler32 (adler32-zlib) [ 9732.793780] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 9733.131417] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 9734.839179] Key type lgssc registered [ 9736.080946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9740.437658] LNet: Added LNI 192.168.203.152@tcp100 [8/256/0/180] [ 9741.232702] LNet: Added LNI 192.168.203.152@tcp101 [8/256/0/180] [ 9742.075761] LNet: Added LNI 192.168.203.152@tcp102 [8/256/0/180] [ 9742.955607] LNet: Added LNI 192.168.203.152@tcp103 [8/256/0/180] [ 9743.685550] LNet: Added LNI 192.168.203.152@tcp104 [8/256/0/180] [ 9744.467228] LNet: Added LNI 192.168.203.152@tcp105 [8/256/0/180] [ 9745.372486] LNet: Added LNI 192.168.203.152@tcp106 [8/256/0/180] [ 9746.120247] LNet: Added LNI 192.168.203.152@tcp107 [8/256/0/180] [ 9746.827934] LNet: Added LNI 192.168.203.152@tcp108 [8/256/0/180] [ 9747.637933] LNet: Added LNI 192.168.203.152@tcp109 [8/256/0/180] [ 9748.378952] LNet: Added LNI 192.168.203.152@tcp110 [8/256/0/180] [ 9749.235781] LNet: Added LNI 192.168.203.152@tcp111 [8/256/0/180] [ 9749.961923] LNet: Added LNI 192.168.203.152@tcp112 [8/256/0/180] [ 9750.899571] LNet: Added LNI 192.168.203.152@tcp113 [8/256/0/180] [ 9751.644677] LNet: Added LNI 192.168.203.152@tcp114 [8/256/0/180] [ 9752.389561] LNet: Added LNI 192.168.203.152@tcp115 [8/256/0/180] [ 9753.250118] LNet: Added LNI 192.168.203.152@tcp116 [8/256/0/180] [ 9754.154683] LNet: Added LNI 192.168.203.152@tcp117 [8/256/0/180] [ 9754.995600] LNet: Added LNI 192.168.203.152@tcp118 [8/256/0/180] [ 9755.797577] LNet: Added LNI 192.168.203.152@tcp119 [8/256/0/180] [ 9756.668938] LNet: Added LNI 192.168.203.152@tcp120 [8/256/0/180] [ 9757.369857] LNet: Added LNI 192.168.203.152@tcp121 [8/256/0/180] [ 9758.209883] LNet: Added LNI 192.168.203.152@tcp122 [8/256/0/180] [ 9758.980364] LNet: Added LNI 192.168.203.152@tcp123 [8/256/0/180] [ 9759.698548] LNet: Added LNI 192.168.203.152@tcp124 [8/256/0/180] [ 9760.302099] LNet: Added LNI 192.168.203.152@tcp125 [8/256/0/180] [ 9761.182863] LNet: Added LNI 192.168.203.152@tcp126 [8/256/0/180] [ 9762.072589] LNet: Added LNI 192.168.203.152@tcp127 [8/256/0/180] [ 9762.866298] LNet: Added LNI 192.168.203.152@tcp128 [8/256/0/180] [ 9763.761588] LNet: Added LNI 192.168.203.152@tcp129 [8/256/0/180] [ 9764.753983] LNet: Added LNI 192.168.203.152@tcp130 [8/256/0/180] [ 9765.839372] LNet: Added LNI 192.168.203.152@tcp131 [8/256/0/180] [ 9766.730970] LNet: Added LNI 192.168.203.152@tcp132 [8/256/0/180] [ 9767.794774] LNet: Added LNI 192.168.203.152@tcp133 [8/256/0/180] [ 9768.904829] LNet: Added LNI 192.168.203.152@tcp134 [8/256/0/180] [ 9769.945894] LNet: Added LNI 192.168.203.152@tcp135 [8/256/0/180] [ 9777.333078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9777.388562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9778.546410] Lustre: 330692:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.152@tcp: can't add NID 192.168.203.152@tcp130: rc = -75 [ 9778.576326] LNetError: 330714:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.152 [ 9778.608381] LNetError: Protocol error connecting to 192.168.203.152@tcp116 at host 192.168.203.152:988: is it running a compatible version of Lustre? [ 9778.619335] LNetError: Skipped 1 previous similar message [ 9778.710668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9778.731795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9778.789586] Lustre: lustre-MDT0000: new disk, initializing [ 9778.853838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9778.874656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9779.236226] LNet: 6 local NIs in recovery (showing 6): 192.168.203.152@tcp121, 192.168.203.152@tcp125, 192.168.203.152@tcp127, 192.168.203.152@tcp129, 192.168.203.152@tcp128, 192.168.203.152@tcp126 [ 9779.239285] LNetError: 328151:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.152 [ 9779.271676] LNetError: 328151:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 9779.290810] LNetError: Protocol error connecting to 192.168.203.152@tcp119 at host 192.168.203.152:988: is it running a compatible version of Lustre? [ 9779.306771] LNetError: Skipped 6 previous similar messages [ 9784.340738] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9793.358983] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9800.381440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9800.558072] Lustre: lustre-OST0000: new disk, initializing [ 9800.565374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9800.571178] Lustre: Skipped 1 previous similar message [ 9800.577185] Lustre: 331713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9800.666911] Lustre: 331718:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.152@tcp: can't add NID 192.168.203.152@tcp130: rc = -75 [ 9800.676683] Lustre: 331718:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9800.714968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9807.039402] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 9808.438196] Lustre: 332125:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.152@tcp: can't add NID 192.168.203.152@tcp130: rc = -75 [ 9808.456582] Lustre: 332125:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 9808.592687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9808.604412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9808.646408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9810.017508] LNetError: 330712:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.152 [ 9810.037526] LNetError: Protocol error connecting to 192.168.203.152@tcp117 at host 192.168.203.152:988: is it running a compatible version of Lustre? [ 9817.185381] LNetError: 330713:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.152 [ 9817.205656] LNetError: Protocol error connecting to 192.168.203.152@tcp116 at host 192.168.203.152:988: is it running a compatible version of Lustre? [ 9817.813671] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9822.310953] LNetError: 328149:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.152 [ 9822.320297] LNetError: 328149:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [ 9822.327819] LNetError: Protocol error connecting to 192.168.203.152@tcp117 at host 192.168.203.152:988: is it running a compatible version of Lustre? [ 9822.342868] LNetError: Skipped 1 previous similar message [ 9829.344302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9829.376184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9830.497309] LNetError: 330714:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.152 [ 9830.505461] LNetError: Protocol error connecting to 192.168.203.152@tcp116 at host 192.168.203.152:988: is it running a compatible version of Lustre? [ 9834.472484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9839.586142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9840.098681] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9840.293706] Lustre: server umount lustre-OST0000 complete [ 9844.383870] Lustre: server umount lustre-MDT0000 complete [ 9848.928880] LNetError: 328151:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.152 [ 9848.935748] LNetError: 328151:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [ 9848.941753] LNetError: Protocol error connecting to 192.168.203.152@tcp117 at host 192.168.203.152:988: is it running a compatible version of Lustre? [ 9848.949407] LNetError: Skipped 4 previous similar messages [ 9854.316944] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 9857.150420] Key type lgssc unregistered [ 9857.578720] LNet: 333420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9857.589317] LNetError: 333420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9858.655492] LNet: Removed LNI 192.168.203.152@tcp [ 9858.665913] LNetError: 333420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9858.690215] LNet: Removed LNI 192.168.203.152@tcp100 [ 9859.752705] LNet: Removed LNI 192.168.203.152@tcp101 [ 9859.773644] LNetError: 333420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9859.783117] LNetError: 333420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9859.794157] LNet: Removed LNI 192.168.203.152@tcp102 [ 9860.831149] LNet: Removed LNI 192.168.203.152@tcp103 [ 9861.857339] LNet: Removed LNI 192.168.203.152@tcp104 [ 9861.862615] LNetError: 333420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9861.868941] LNetError: 333420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9861.874723] LNet: Removed LNI 192.168.203.152@tcp105 [ 9862.879421] LNet: Removed LNI 192.168.203.152@tcp106 [ 9863.903227] LNet: Removed LNI 192.168.203.152@tcp107 [ 9864.927909] LNet: Removed LNI 192.168.203.152@tcp108 [ 9865.951169] LNet: Removed LNI 192.168.203.152@tcp109 [ 9865.954186] LNetError: 333420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9865.964487] LNetError: 333420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9865.977838] LNet: Removed LNI 192.168.203.152@tcp110 [ 9867.039268] LNet: Removed LNI 192.168.203.152@tcp111 [ 9868.063359] LNet: Removed LNI 192.168.203.152@tcp112 [ 9869.087523] LNet: Removed LNI 192.168.203.152@tcp113 [ 9870.111303] LNet: Removed LNI 192.168.203.152@tcp114 [ 9871.135140] LNet: Removed LNI 192.168.203.152@tcp115 [ 9872.159200] LNet: Removed LNI 192.168.203.152@tcp116 [ 9873.187071] LNet: Removed LNI 192.168.203.152@tcp117 [ 9874.207165] LNet: Removed LNI 192.168.203.152@tcp118 [ 9874.209421] LNetError: 333420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9874.219071] LNetError: 333420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 9874.240175] LNet: Removed LNI 192.168.203.152@tcp119 [ 9875.295480] LNet: Removed LNI 192.168.203.152@tcp120 [ 9876.319154] LNet: Removed LNI 192.168.203.152@tcp121 [ 9877.343427] LNet: Removed LNI 192.168.203.152@tcp122 [ 9878.369478] LNet: Removed LNI 192.168.203.152@tcp123 [ 9879.391270] LNet: Removed LNI 192.168.203.152@tcp124 [ 9880.415834] LNet: Removed LNI 192.168.203.152@tcp125 [ 9881.439145] LNet: Removed LNI 192.168.203.152@tcp126 [ 9882.463292] LNet: Removed LNI 192.168.203.152@tcp127 [ 9883.487115] LNet: Removed LNI 192.168.203.152@tcp128 [ 9884.511249] LNet: Removed LNI 192.168.203.152@tcp129 [ 9884.522225] LNet: Removed LNI 192.168.203.152@tcp130 [ 9884.530296] LNet: Removed LNI 192.168.203.152@tcp131 [ 9884.536799] LNet: Removed LNI 192.168.203.152@tcp132 [ 9884.544445] LNet: Removed LNI 192.168.203.152@tcp133 [ 9884.550213] LNet: Removed LNI 192.168.203.152@tcp134 [ 9884.566219] LNet: Removed LNI 192.168.203.152@tcp135 [ 9885.413561] Key type .llcrypt unregistered [ 9885.415659] Key type ._llcrypt unregistered [ 9904.259490] Key type ._llcrypt registered [ 9904.262911] Key type .llcrypt registered [ 9904.376368] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 9916.873763] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 9917.813808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9917.837560] alg: No test for adler32 (adler32-zlib) [ 9918.948600] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 9919.155657] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 9920.863181] Key type lgssc registered [ 9922.115370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9950.055706] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:41:52 (1785087712) [ 9965.509407] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 9968.410349] Key type lgssc unregistered [ 9968.760588] LNet: 336922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9968.784427] LNetError: 336922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9968.818619] LNet: Removed LNI 192.168.203.152@tcp [ 9969.645534] Key type .llcrypt unregistered [ 9969.647078] Key type ._llcrypt unregistered [ 9986.140251] Key type ._llcrypt registered [ 9986.142572] Key type .llcrypt registered [ 9986.242303] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 9987.343252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9987.402489] alg: No test for adler32 (adler32-zlib) [ 9988.379798] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [ 9988.560707] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 9990.207311] Key type lgssc registered [ 9991.293079] Lustre: Echo OBD driver; http://www.lustre.org/ [10000.257235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10000.284534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10001.653801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10001.702465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10001.817383] Lustre: lustre-MDT0000: new disk, initializing [10002.022267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10002.049801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10007.373457] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10017.756516] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10024.999657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10025.237546] Lustre: lustre-OST0000: new disk, initializing [10025.240329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10025.245403] Lustre: Skipped 1 previous similar message [10025.251485] Lustre: 339128:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10025.348762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10031.224875] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10034.253153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10034.264111] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10034.296522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10039.867436] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10044.470460] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing lnet_if_list [10050.314052] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [10050.361292] LNet: Added LNI 192.168.203.152@tcp10 [8/256/0/180] [10057.027285] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [10057.069376] LNet: Added LNI 192.168.203.152@tcp11 [8/256/0/180] [10061.212576] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [10061.249316] LNet: Added LNI 192.168.203.152@tcp12 [8/256/0/180] [10065.375898] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [10065.403598] LNet: Added LNI 192.168.203.152@tcp13 [8/256/0/180] [10069.802673] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [10069.852318] LNet: Added LNI 192.168.203.152@tcp14 [8/256/0/180] [10075.067871] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [10075.117069] LNet: Added LNI 192.168.203.152@tcp15 [8/256/0/180] [10079.874152] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [10079.905990] LNet: Added LNI 192.168.203.152@tcp16 [8/256/0/180] [10084.663216] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10084.714693] LNet: Added LNI 192.168.203.152@tcp17 [8/256/0/180] [10089.391118] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [10089.433966] LNet: Added LNI 192.168.203.152@tcp18 [8/256/0/180] [10094.065665] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [10094.101676] LNet: Added LNI 192.168.203.152@tcp19 [8/256/0/180] [10099.230733] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [10099.261131] LNet: Added LNI 192.168.203.152@tcp20 [8/256/0/180] [10104.528081] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [10104.572174] LNet: Added LNI 192.168.203.152@tcp21 [8/256/0/180] [10108.952479] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [10108.975498] LNet: Added LNI 192.168.203.152@tcp22 [8/256/0/180] [10114.014596] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [10114.050888] LNet: Added LNI 192.168.203.152@tcp23 [8/256/0/180] [10119.341083] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [10119.370614] LNet: Added LNI 192.168.203.152@tcp24 [8/256/0/180] [10126.120531] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [10126.176811] LNet: Added LNI 192.168.203.152@tcp25 [8/256/0/180] [10133.996986] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [10134.051760] LNet: Added LNI 192.168.203.152@tcp26 [8/256/0/180] [10142.011482] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [10142.074127] LNet: Added LNI 192.168.203.152@tcp27 [8/256/0/180] [10148.632319] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [10148.697815] LNet: Added LNI 192.168.203.152@tcp28 [8/256/0/180] [10154.249308] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [10154.270465] LNet: Added LNI 192.168.203.152@tcp29 [8/256/0/180] [10160.057298] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [10160.073095] LNet: Added LNI 192.168.203.152@tcp30 [8/256/0/180] [10165.829306] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [10165.849658] LNet: Added LNI 192.168.203.152@tcp31 [8/256/0/180] [10171.922303] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [10171.948494] LNet: Added LNI 192.168.203.152@tcp32 [8/256/0/180] [10178.445316] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [10178.469520] LNet: Added LNI 192.168.203.152@tcp33 [8/256/0/180] [10185.021097] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10185.050361] LNet: Added LNI 192.168.203.152@tcp34 [8/256/0/180] [10192.153307] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10192.182220] LNet: Added LNI 192.168.203.152@tcp35 [8/256/0/180] [10198.882669] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10198.928566] LNet: Added LNI 192.168.203.152@tcp36 [8/256/0/180] [10205.935697] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10205.971298] LNet: Added LNI 192.168.203.152@tcp37 [8/256/0/180] [10213.511168] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10213.553733] LNet: Added LNI 192.168.203.152@tcp38 [8/256/0/180] [10220.141366] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10220.166976] LNet: Added LNI 192.168.203.152@tcp39 [8/256/0/180] [10227.596327] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10227.626975] LNet: Added LNI 192.168.203.152@tcp40 [8/256/0/180] [10234.767728] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10234.791365] LNet: Added LNI 192.168.203.152@tcp41 [8/256/0/180] [10241.899606] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10241.944462] LNet: Added LNI 192.168.203.152@tcp42 [8/256/0/180] [10249.914720] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10249.941701] LNet: Added LNI 192.168.203.152@tcp43 [8/256/0/180] [10257.905519] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10257.946916] LNet: Added LNI 192.168.203.152@tcp44 [8/256/0/180] [10266.148646] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10266.168851] LNet: Added LNI 192.168.203.152@tcp45 [8/256/0/180] [10273.601806] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10273.633349] LNet: Added LNI 192.168.203.152@tcp46 [8/256/0/180] [10280.793404] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10280.834589] LNet: Added LNI 192.168.203.152@tcp47 [8/256/0/180] [10288.340511] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10288.373756] LNet: Added LNI 192.168.203.152@tcp48 [8/256/0/180] [10295.221408] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10295.241456] LNet: Added LNI 192.168.203.152@tcp49 [8/256/0/180] [10303.129384] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10303.151576] LNet: Added LNI 192.168.203.152@tcp50 [8/256/0/180] [10320.865517] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10321.057629] Lustre: server umount lustre-MDT0000 complete [10339.446607] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10341.355102] Lustre: 337405:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785088090/real 1785088090] req@000000005dffd423 x1871800175756416/t0(0) o400->MGC192.168.203.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785088106 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10341.384513] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10348.374510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10351.586540] LustreError: 337403:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005b79b73c x1871800175757312/t0(0) o250->MGC192.168.203.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10352.034680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10357.255961] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10360.303085] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10364.713235] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10385.894239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10385.907331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10389.481608] Lustre: server umount lustre-OST0000 complete [10393.002996] Lustre: server umount lustre-MDT0000 complete [10403.706721] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [10406.500687] Key type lgssc unregistered [10406.831231] LNet: 364837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10406.837682] LNetError: 364837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10406.846782] LNet: Removed LNI 192.168.203.152@tcp [10406.852541] LNet: Removed LNI 192.168.203.152@tcp10 [10406.858248] LNet: Removed LNI 192.168.203.152@tcp11 [10406.865952] LNet: Removed LNI 192.168.203.152@tcp12 [10406.870791] LNet: Removed LNI 192.168.203.152@tcp13 [10406.875766] LNet: Removed LNI 192.168.203.152@tcp14 [10406.879632] LNet: Removed LNI 192.168.203.152@tcp15 [10406.883624] LNet: Removed LNI 192.168.203.152@tcp16 [10406.888887] LNet: Removed LNI 192.168.203.152@tcp17 [10406.892470] LNet: Removed LNI 192.168.203.152@tcp18 [10406.895737] LNet: Removed LNI 192.168.203.152@tcp19 [10406.901985] LNet: Removed LNI 192.168.203.152@tcp20 [10406.906481] LNet: Removed LNI 192.168.203.152@tcp21 [10406.910942] LNet: Removed LNI 192.168.203.152@tcp22 [10406.917892] LNet: Removed LNI 192.168.203.152@tcp23 [10406.921437] LNet: Removed LNI 192.168.203.152@tcp24 [10406.923579] LNet: Removed LNI 192.168.203.152@tcp25 [10406.926148] LNet: Removed LNI 192.168.203.152@tcp26 [10406.929369] LNet: Removed LNI 192.168.203.152@tcp27 [10406.932286] LNet: Removed LNI 192.168.203.152@tcp28 [10406.935266] LNet: Removed LNI 192.168.203.152@tcp29 [10406.938276] LNet: Removed LNI 192.168.203.152@tcp30 [10406.941350] LNet: Removed LNI 192.168.203.152@tcp31 [10406.945694] LNet: Removed LNI 192.168.203.152@tcp32 [10406.949805] LNet: Removed LNI 192.168.203.152@tcp33 [10406.953243] LNet: Removed LNI 192.168.203.152@tcp34 [10406.956766] LNet: Removed LNI 192.168.203.152@tcp35 [10406.958754] LNet: Removed LNI 192.168.203.152@tcp36 [10406.962065] LNet: Removed LNI 192.168.203.152@tcp37 [10406.964695] LNet: Removed LNI 192.168.203.152@tcp38 [10406.967277] LNet: Removed LNI 192.168.203.152@tcp39 [10406.969534] LNet: Removed LNI 192.168.203.152@tcp40 [10406.972433] LNet: Removed LNI 192.168.203.152@tcp41 [10406.975017] LNet: Removed LNI 192.168.203.152@tcp42 [10406.977640] LNet: Removed LNI 192.168.203.152@tcp43 [10406.980520] LNet: Removed LNI 192.168.203.152@tcp44 [10406.983292] LNet: Removed LNI 192.168.203.152@tcp45 [10406.986078] LNet: Removed LNI 192.168.203.152@tcp46 [10406.988831] LNet: Removed LNI 192.168.203.152@tcp47 [10406.991248] LNet: Removed LNI 192.168.203.152@tcp48 [10406.994191] LNet: Removed LNI 192.168.203.152@tcp49 [10407.022215] LNet: Removed LNI 192.168.203.152@tcp50 [10407.918353] Key type .llcrypt unregistered [10407.920188] Key type ._llcrypt unregistered [10423.388548] Key type ._llcrypt registered [10423.391291] Key type .llcrypt registered [10423.506925] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [10437.135714] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10437.950827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10437.993891] alg: No test for adler32 (adler32-zlib) [10439.069245] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [10439.286431] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [10440.927255] Key type lgssc registered [10441.955909] Lustre: Echo OBD driver; http://www.lustre.org/ [10468.743321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10468.763383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10469.996303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10470.036549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10470.154378] Lustre: lustre-MDT0000: new disk, initializing [10470.293934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10470.324959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10474.281730] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10483.165653] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10489.898059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10490.132996] Lustre: *** cfs_fail_loc=90f, val=0*** [10490.196860] Lustre: lustre-OST0000: new disk, initializing [10490.201503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10490.207279] Lustre: Skipped 1 previous similar message [10490.213613] Lustre: 368660:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10490.300170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10496.034542] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10500.733847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10500.742215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10500.771416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10504.833340] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10510.820353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10510.822051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10510.843680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10515.938022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10516.864948] Lustre: server umount lustre-OST0000 complete [10520.669356] Lustre: server umount lustre-MDT0000 complete [10531.439304] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [10534.058116] Key type lgssc unregistered [10534.339328] LNet: 370355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10534.347235] LNetError: 370355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10535.399256] LNet: Removed LNI 192.168.203.152@tcp [10536.434667] Key type .llcrypt unregistered [10536.440232] Key type ._llcrypt unregistered [10554.371722] Key type ._llcrypt registered [10554.373539] Key type .llcrypt registered [10554.470427] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10555.225084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10555.275080] alg: No test for adler32 (adler32-zlib) [10556.322914] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [10556.498660] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [10558.159170] Key type lgssc registered [10559.176894] Lustre: Echo OBD driver; http://www.lustre.org/ [10566.482494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10566.497675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10567.866988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10571.938317] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10578.911981] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10585.195922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10585.616485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10592.431556] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10599.640542] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10609.705417] loop0: detected capacity change from 0 to 204800000 [10609.946033] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10609.962360] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10609.979325] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10615.568158] loop0: detected capacity change from 0 to 204800000 [10615.744451] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10615.760983] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10615.778398] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10620.016883] loop0: detected capacity change from 0 to 204800000 [10624.709262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10624.873526] Lustre: fs15246-OST0000: new disk, initializing [10624.880780] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10624.893233] Lustre: 374310:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10624.955693] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10629.594543] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10638.190859] loop1: detected capacity change from 0 to 204800000 [10643.399082] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10643.521538] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10643.611758] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10643.737602] Lustre: fs15246-MDT0000: new disk, initializing [10643.923153] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10643.955959] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10644.064510] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10645.092474] Lustre: 375432:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000aaa53a05 x1871800771302144/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785088426 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10648.292764] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10654.216067] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10654.234968] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10654.292515] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10671.078688] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10671.087924] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10673.552264] Lustre: server umount fs15246-MDT0000 complete [10681.316603] LustreError: 371617:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10681.350381] LustreError: 371617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10681.989738] Lustre: server umount fs15246-OST0000 complete [10695.136354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10695.153439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10695.167757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10696.675545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10698.917818] Lustre: server umount lustre-OST0000 complete [10702.626274] Lustre: server umount lustre-MDT0000 complete [10712.281843] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [10714.364880] Key type lgssc unregistered [10714.562667] LNet: 378325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10714.567050] LNetError: 378325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10714.582540] LNet: Removed LNI 192.168.203.152@tcp [10715.274213] Key type .llcrypt unregistered [10715.276455] Key type ._llcrypt unregistered [10730.304431] Key type ._llcrypt registered [10730.306713] Key type .llcrypt registered [10730.410599] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10731.167847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10731.212770] alg: No test for adler32 (adler32-zlib) [10732.298743] Lustre: Lustre: Build Version: 2.17.54_162_gf7f7620 [10732.496351] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [10734.167485] Key type lgssc registered [10735.273894] Lustre: Echo OBD driver; http://www.lustre.org/ [10766.306724] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [10775.206215] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10800.235980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10800.257930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10801.443847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10801.476239] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10801.570641] Lustre: lustre-MDT0000: new disk, initializing [10801.638524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10801.656451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10805.813945] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10813.738111] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10819.348509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10819.562522] Lustre: lustre-OST0000: new disk, initializing [10819.567672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10819.572794] Lustre: Skipped 1 previous similar message [10819.578334] Lustre: 383352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10819.631092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10824.957767] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [10827.834318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10827.841107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10827.861184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10832.565503] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10836.215744] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10836.414887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10843.105522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10843.113433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10843.127619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10848.224338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10852.319823] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10852.462539] Lustre: server umount lustre-OST0000 complete [10855.208898] Lustre: server umount lustre-MDT0000 complete [10878.366212] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [10887.988433] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [10909.231937] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10681 sec ========= 13:57:51 (1785088671) [10911.100967] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:57:53 (1785088673) === [10914.259222] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:57:56 (1785088676) === [10941.715418] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [10944.132479] Key type lgssc unregistered [10944.382886] LNet: 388702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10944.392625] LNetError: 388702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10944.417797] LNet: Removed LNI 192.168.203.152@tcp [10945.269275] Key type .llcrypt unregistered [10945.272574] Key type ._llcrypt unregistered