[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 511623434 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003228] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008574] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011150] LSM: Security Framework initializing [ 0.013044] Yama: becoming mindful. [ 0.014000] SELinux: Initializing. [ 0.014092] *** VALIDATE selinux *** [ 0.022754] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027524] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031120] *** VALIDATE tmpfs *** [ 0.032282] *** VALIDATE proc *** [ 0.034235] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.037102] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041007] Speculative Store Bypass: Vulnerable [ 0.044474] debug: unmapping init [mem 0xffffffff94659000-0xffffffff94660fff] [ 0.046305] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047840] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048021] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052013] ... max period: 00007fffffffffff [ 0.053013] ... fixed-purpose events: 3 [ 0.054010] ... event mask: 000000070000000f [ 0.055288] rcu: Hierarchical SRCU implementation. [ 0.057673] smp: Bringing up secondary CPUs ... [ 0.058597] x86: Booting SMP configuration: [ 0.059025] .... node #0, CPUs: #1 #2 #3 [ 0.062706] smp: Brought up 1 node, 4 CPUs [ 0.063999] smpboot: Max logical packages: 1 [ 0.065012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.133125] node 0 deferred pages initialised in 66ms [ 0.138010] devtmpfs: initialized [ 0.139240] x86/mm: Memory block size: 128MB [ 0.141913] gcov: version magic: 0x41383552 [ 0.145084] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147297] pinctrl core: initialized pinctrl subsystem [ 0.148197] [ 0.148778] ************************************************************* [ 0.149016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150015] ** ** [ 0.151012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.152017] ** ** [ 0.153018] ** This means that this kernel is built to expose internal ** [ 0.154016] ** IOMMU data structures, which may compromise security on ** [ 0.155014] ** your system. ** [ 0.156016] ** ** [ 0.157016] ** If you see this message and you are not debugging the ** [ 0.158015] ** kernel, report this immediately to your vendor! ** [ 0.159014] ** ** [ 0.160018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161015] ************************************************************* [ 0.162883] NET: Registered protocol family 16 [ 0.163557] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.164061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.165049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.166376] cpuidle: using governor menu [ 0.169845] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.173557] PCI: Using configuration type 1 for base access [ 0.176124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185799] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188068] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192265] cryptd: max_cpu_qlen set to 1000 [ 0.195196] ACPI: Added _OSI(Module Device) [ 0.197011] ACPI: Added _OSI(Processor Device) [ 0.198137] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.200012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.205883] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.212574] ACPI: Interpreter enabled [ 0.213061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214012] ACPI: Using IOAPIC for interrupt routing [ 0.215107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216503] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.224895] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.228042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.230016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.239431] acpiphp: Slot [2] registered [ 0.241106] acpiphp: Slot [5] registered [ 0.242111] acpiphp: Slot [6] registered [ 0.244101] acpiphp: Slot [7] registered [ 0.245105] acpiphp: Slot [8] registered [ 0.247095] acpiphp: Slot [9] registered [ 0.248110] acpiphp: Slot [10] registered [ 0.250134] acpiphp: Slot [3] registered [ 0.251069] acpiphp: Slot [4] registered [ 0.253128] acpiphp: Slot [11] registered [ 0.254064] acpiphp: Slot [12] registered [ 0.254998] acpiphp: Slot [13] registered [ 0.256085] acpiphp: Slot [14] registered [ 0.258142] acpiphp: Slot [15] registered [ 0.259201] acpiphp: Slot [16] registered [ 0.261087] acpiphp: Slot [17] registered [ 0.263113] acpiphp: Slot [18] registered [ 0.264070] acpiphp: Slot [19] registered [ 0.265060] acpiphp: Slot [20] registered [ 0.266062] acpiphp: Slot [21] registered [ 0.267052] acpiphp: Slot [22] registered [ 0.268034] acpiphp: Slot [23] registered [ 0.269056] acpiphp: Slot [24] registered [ 0.270085] acpiphp: Slot [25] registered [ 0.272083] acpiphp: Slot [26] registered [ 0.273106] acpiphp: Slot [27] registered [ 0.274083] acpiphp: Slot [28] registered [ 0.274944] acpiphp: Slot [29] registered [ 0.276079] acpiphp: Slot [30] registered [ 0.277069] acpiphp: Slot [31] registered [ 0.277943] PCI host bridge to bus 0000:00 [ 0.279014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.280015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.282013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.283016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.285012] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.287029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.290855] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.293454] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.304954] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.307517] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.309015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.311010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.312014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.315494] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.317552] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.319033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.322536] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.329015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.340017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.345012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.351099] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.358022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.369018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.386018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.394856] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.401021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.413016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.437017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.451040] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.459018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.469019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.494016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.504093] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.515017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.521014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.537016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.546420] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.555016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.562016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.577018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.588308] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.594017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.605016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.626020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.635872] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.639397] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.641324] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.643430] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.646235] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.651125] iommu: Default domain type: Passthrough [ 0.652415] SCSI subsystem initialized [ 0.654156] ACPI: bus type USB registered [ 0.656160] usbcore: registered new interface driver usbfs [ 0.658067] usbcore: registered new interface driver hub [ 0.660092] usbcore: registered new device driver usb [ 0.662181] pps_core: LinuxPPS API ver. 1 registered [ 0.664010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.667074] PTP clock support registered [ 0.670032] EDAC MC: Ver: 3.0.0 [ 0.671625] PCI: Using ACPI for IRQ routing [ 0.672872] NetLabel: Initializing [ 0.673009] NetLabel: domain hash size = 128 [ 0.674005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.675093] NetLabel: unlabeled traffic allowed by default [ 0.676007] vgaarb: loaded [ 0.677321] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.679014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.686111] clocksource: Switched to clocksource kvm-clock [ 0.786787] VFS: Disk quotas dquot_6.6.0 [ 0.788038] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.790048] *** VALIDATE ramfs *** [ 0.791408] *** VALIDATE hugetlbfs *** [ 0.792469] pnp: PnP ACPI init [ 0.794938] pnp: PnP ACPI: found 6 devices [ 0.810094] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.812987] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.814942] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.817417] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.820148] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.822794] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.826213] NET: Registered protocol family 2 [ 0.829581] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.834505] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.838075] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.843269] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.845767] TCP: Hash tables configured (established 65536 bind 65536) [ 0.847631] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.849758] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.851614] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.853884] NET: Registered protocol family 1 [ 0.855777] RPC: Registered named UNIX socket transport module. [ 0.857872] RPC: Registered udp transport module. [ 0.858943] RPC: Registered tcp transport module. [ 0.859993] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.861517] NET: Registered protocol family 44 [ 0.862695] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.864381] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.865756] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.867623] PCI: CLS 0 bytes, default 64 [ 0.868757] Unpacking initramfs... [ 2.311859] debug: unmapping init [mem 0xffff945cbcc54000-0xffff945cbffbffff] [ 2.314927] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.317206] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.320204] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.843851] Initialise system trusted keyrings [ 2.845635] Key type blacklist registered [ 2.847481] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.856606] zbud: loaded [ 2.859247] *** VALIDATE nfs *** [ 2.860627] *** VALIDATE nfs4 *** [ 2.862248] pstore: using deflate compression [ 2.866655] Platform Keyring initialized [ 2.955942] NET: Registered protocol family 38 [ 2.957238] Key type asymmetric registered [ 2.957896] Asymmetric key parser 'x509' registered [ 2.959124] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.961072] io scheduler mq-deadline registered [ 2.961776] io scheduler kyber registered [ 2.962514] io scheduler bfq registered [ 2.963389] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.965096] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.966942] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.968631] ACPI: Power Button [PWRF] [ 2.972759] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.976751] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.000664] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.013517] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.047685] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.072272] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.098095] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.102172] Non-volatile memory driver v1.3 [ 3.103323] Linux agpgart interface v0.103 [ 3.126105] virtio_blk virtio1: [vda] 134360 512-byte logical blocks (68.8 MB/65.6 MiB) [ 3.128688] vda: detected capacity change from 0 to 68792320 [ 3.163402] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.166032] vdb: detected capacity change from 0 to 1073741824 [ 3.206768] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.210083] vdc: detected capacity change from 0 to 2621440000 [ 3.229068] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.231966] vdd: detected capacity change from 0 to 2621440000 [ 3.244947] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.246823] vde: detected capacity change from 0 to 4294967296 [ 3.263271] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.265548] vdf: detected capacity change from 0 to 4294967296 [ 3.273167] libphy: Fixed MDIO Bus: probed [ 3.282206] usbcore: registered new interface driver usbserial_generic [ 3.284369] usbserial: USB Serial support registered for generic [ 3.286074] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.288586] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.289606] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.291136] mousedev: PS/2 mouse device common for all mice [ 3.292651] rtc_cmos 00:05: RTC can wake from S4 [ 3.294336] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.296731] rtc_cmos 00:05: registered as rtc0 [ 3.298569] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.299031] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.302753] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.303277] intel_pstate: CPU model not supported [ 3.306625] hid: raw HID events driver (C) Jiri Kosina [ 3.307463] usbcore: registered new interface driver usbhid [ 3.308283] usbhid: USB HID core driver [ 3.308979] drop_monitor: Initializing network drop monitor service [ 3.309988] Initializing XFRM netlink socket [ 3.311056] NET: Registered protocol family 10 [ 3.312681] Segment Routing with IPv6 [ 3.313346] NET: Registered protocol family 17 [ 3.314305] mpls_gso: MPLS GSO support [ 3.317690] RAS: Correctable Errors collector initialized. [ 3.318880] AVX version of gcm_enc/dec engaged. [ 3.319789] AES CTR mode by8 optimization enabled [ 3.369591] sched_clock: Marking stable (3369574465, 0)->(4327444372, -957869907) [ 3.372242] registered taskstats version 1 [ 3.373372] Loading compiled-in X.509 certificates [ 3.374472] zswap: loaded using pool lzo/zbud [ 3.390625] Key type big_key registered [ 3.398787] Key type encrypted registered [ 3.399789] ima: No TPM chip found, activating TPM-bypass! [ 3.400949] ima: Allocated hash algorithm: sha1 [ 3.401791] ima: No architecture policies found [ 3.403098] evm: Initialising EVM extended attributes: [ 3.404194] evm: security.selinux [ 3.404894] evm: security.ima [ 3.405618] evm: security.capability [ 3.406424] evm: HMAC attrs: 0x1 [ 3.408084] rtc_cmos 00:05: setting system clock to 2026-01-30 06:33:57 UTC (1769754837) [ 3.412318] debug: unmapping init [mem 0xffffffff95603000-0xffffffff957fffff] [ 3.414301] debug: unmapping init [mem 0xffffffff94382000-0xffffffff94658fff] [ 3.423048] Write protecting the kernel read-only data: 28672k [ 3.425243] debug: unmapping init [mem 0xffffffff92a03000-0xffffffff92bfffff] [ 3.426989] debug: unmapping init [mem 0xffffffff93314000-0xffffffff933fffff] [ 3.450091] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.455743] systemd[1]: Detected virtualization kvm. [ 3.456934] systemd[1]: Detected architecture x86-64. [ 3.458067] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.480663] systemd[1]: No hostname configured. [ 3.482222] systemd[1]: Set hostname to . [ 3.483583] random: systemd: uninitialized urandom read (16 bytes read) [ 3.485369] systemd[1]: Initializing machine ID from random generator. [ 3.607771] random: systemd: uninitialized urandom read (16 bytes read) [ 3.609461] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.611786] random: systemd: uninitialized urandom read (16 bytes read) [ 3.613393] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.615593] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.228146] device-mapper: uevent: version 1.0.3 [ 4.229480] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.917592] virtio_net virtio0 ens2: renamed from eth0 [ 4.960112] scsi host0: ata_piix [ 4.962817] random: fast init done [ 5.016240] scsi host1: ata_piix [ 5.018172] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.020724] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.753181] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.910914] random: crng init done [ 9.912395] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.367777] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.567655] printk: systemd: 25 output lines suppressed due to ratelimiting [ 14.732644] SELinux: Disabled at runtime. [ 14.941923] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.969409] systemd[1]: Detected virtualization kvm. [ 14.977621] systemd[1]: Detected architecture x86-64. [ 15.004142] hrtimer: interrupt took 2280977 ns Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.247706] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.257058] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.275689] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.286546] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.294649] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.327156] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.376723] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 17.926780] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 19.732186] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.291844] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.514252] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.091734] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.133889] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 28.288445] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (11s / no limit)[ 29.274715] NFS: Registering the id_resolver key type [ 29.281359] Key type id_resolver registered [ 29.286283] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg121-server login: [ 86.611350] libcfs: loading out-of-tree module taints kernel. [ 86.736846] Key type ._llcrypt registered [ 86.739568] Key type .llcrypt registered [ 86.897744] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 110.036678] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 111.731355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 111.784233] alg: No test for adler32 (adler32-zlib) [ 113.273631] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 114.277554] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 116.207173] Key type lgssc registered [ 118.716577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 139.854398] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 173.027933] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 187.395894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 187.445776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.003691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.053410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 189.155845] Lustre: lustre-MDT0000: new disk, initializing [ 189.312610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.357878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.735320] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.002755] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 210.225486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.475154] Lustre: lustre-OST0000: new disk, initializing [ 210.478859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.489481] Lustre: Skipped 1 previous similar message [ 210.530797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.805223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 213.833629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 213.865865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 216.088982] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.321105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.478263] Lustre: lustre-OST0001: new disk, initializing [ 232.483070] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 232.536161] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 238.705745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 238.710189] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 238.780356] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 242.348229] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.142938] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 263.095120] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 275.289731] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing check_logdir /tmp/testlogs/ [ 280.314435] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing yml_node [ 283.834801] Lustre: DEBUG MARKER: Client: 2.17.50.86 [ 286.500421] Lustre: DEBUG MARKER: MDS: 2.17.50.86 [ 289.097499] Lustre: DEBUG MARKER: OSS: 2.17.50.86 [ 290.911798] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jan 30 01:38:44 EST 2026 [ 308.843958] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 310.303470] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 328.671753] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.678801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.682886] Lustre: Skipped 1 previous similar message [ 328.704893] Lustre: Skipped 1 previous similar message [ 332.423915] LustreError: 10698:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 332.818412] Lustre: server umount lustre-MDT0000 complete [ 336.738408] LustreError: 5925:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755170 with bad export cookie 4771646716014836487 [ 336.743325] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 336.748985] LustreError: 5925:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 336.790894] LustreError: 10901:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 336.793518] LustreError: 10901:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 336.836374] Lustre: server umount lustre-OST0000 complete [ 340.969152] LustreError: 11103:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 340.975605] LustreError: 11103:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 341.069896] Lustre: server umount lustre-OST0001 complete [ 348.596791] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 357.463325] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 387.852073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.182032] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 388.220692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 388.296814] Lustre: lustre-MDT0000: new disk, initializing [ 388.395183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.421570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 393.090686] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 400.794775] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 407.807971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 408.095738] Lustre: lustre-OST0000: new disk, initializing [ 408.101235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 408.109230] Lustre: Skipped 1 previous similar message [ 408.215231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.565084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 409.574807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 409.642247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 414.906236] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 422.536543] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 427.516586] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 427.803774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 430.055147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 430.068854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.122273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.170290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.655685] LustreError: 14832:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 435.659706] LustreError: 14832:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 435.821691] Lustre: server umount lustre-OST0000 complete [ 439.870639] LustreError: 15034:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 439.875482] LustreError: 15034:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 440.190301] Lustre: server umount lustre-MDT0000 complete [ 444.868217] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:41:17 (1769755277) [ 447.538914] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 449.206872] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:41:22 (1769755282) [ 456.201133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.732603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.172587] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 467.001304] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 474.062293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 474.559952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.853424] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 489.607931] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 498.153482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 498.154914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.208466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.265235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.793339] LustreError: 17146:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 503.802237] LustreError: 17146:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 503.906903] Lustre: server umount lustre-OST0000 complete [ 508.474509] Lustre: server umount lustre-MDT0000 complete [ 519.583947] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 522.563078] Key type lgssc unregistered [ 522.916719] LNet: 17881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 522.930888] LNetError: 17881:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 522.966351] LNet: Removed LNI 192.168.201.121@tcp [ 524.016218] Key type .llcrypt unregistered [ 524.021359] Key type ._llcrypt unregistered [ 554.174691] Key type ._llcrypt registered [ 554.176382] Key type .llcrypt registered [ 554.366774] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 555.728941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 555.786502] alg: No test for adler32 (adler32-zlib) [ 557.090555] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 557.449317] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 559.175682] Key type lgssc registered [ 560.675853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 570.422290] loop: module loaded [ 570.565180] loop0: detected capacity change from 0 to 204800000 [ 570.877393] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 570.893474] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 571.275831] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 578.603702] loop0: detected capacity change from 0 to 204800000 [ 578.890735] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 578.897643] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 578.904495] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 586.210579] loop0: detected capacity change from 0 to 204800000 [ 592.372396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 592.400178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.776276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 593.930793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 594.121849] Lustre: lustre-MDT0000: new disk, initializing [ 594.267256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.298955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 599.692960] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 610.407091] loop1: detected capacity change from 0 to 204800000 [ 616.733908] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 617.154037] Lustre: lustre-OST07c6: new disk, initializing [ 617.159440] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 617.168165] Lustre: Skipped 1 previous similar message [ 617.276821] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 623.796468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 623.802900] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 623.841991] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 624.974930] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 630.226933] Lustre: 20033:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 630.243832] Lustre: 20033:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 630.253272] Lustre: 20033:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 630.264459] Lustre: 20033:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 630.274384] Lustre: 20033:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 630.282541] Lustre: 20033:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 648.706472] Lustre: 20033:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 648.712865] Lustre: 20033:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 648.717914] Lustre: 20033:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 648.723540] Lustre: 20033:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 648.734957] Lustre: 20033:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 648.740528] Lustre: 20033:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 653.970987] Lustre: 20033:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 653.986126] Lustre: 20033:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 653.999062] Lustre: 20033:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 654.014149] Lustre: 20033:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 654.029502] Lustre: 20033:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 654.046526] Lustre: 20033:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 654.058968] Lustre: 20033:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 654.075452] Lustre: 20033:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 654.085469] Lustre: 20033:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 654.100378] Lustre: 20033:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 654.109841] Lustre: 20033:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 654.119832] Lustre: 20033:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.599645] Lustre: 20033:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 659.625656] Lustre: 20033:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 659.644658] Lustre: 20033:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 659.653054] Lustre: 20033:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.661879] Lustre: 20033:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 659.679897] Lustre: 20033:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.690045] Lustre: 20033:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 659.699915] Lustre: 20033:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.714442] Lustre: 20033:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 659.723064] Lustre: 20033:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.731227] Lustre: 20033:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 659.738858] Lustre: 20033:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.893082] Lustre: 21527:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 664.906261] Lustre: 21527:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 664.915322] Lustre: 21527:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 664.919922] Lustre: 21527:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.936816] Lustre: 21527:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 664.945929] Lustre: 21527:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.954467] Lustre: 21527:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 664.960968] Lustre: 21527:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.968871] Lustre: 21527:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 664.975186] Lustre: 21527:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.982767] Lustre: 21527:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 664.993717] Lustre: 21527:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 675.630505] Lustre: 20035:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 675.648374] Lustre: 20035:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 3 previous similar messages [ 675.661384] Lustre: 20035:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 675.682647] Lustre: 20035:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 675.702070] Lustre: 20035:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 675.711392] Lustre: 20035:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 675.727494] Lustre: 20035:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 675.752318] Lustre: 20035:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 675.760155] Lustre: 20035:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 675.782260] Lustre: 20035:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 675.792143] Lustre: 20035:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 675.804564] Lustre: 20035:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 726.498083] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 726.513707] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 726.565654] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 728.184504] LustreError: 22214:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 728.297308] Lustre: server umount lustre-OST07c6 complete [ 736.599936] LustreError: 22715:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 736.606533] LustreError: 22715:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 736.781628] Lustre: server umount lustre-MDT0000 complete [ 751.551195] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 754.210501] Key type lgssc unregistered [ 754.486277] LNet: 23548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.499386] LNetError: 23548:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.542409] LNet: Removed LNI 192.168.201.121@tcp [ 755.651933] Key type .llcrypt unregistered [ 755.656330] Key type ._llcrypt unregistered [ 773.428534] Key type ._llcrypt registered [ 773.430809] Key type .llcrypt registered [ 773.529780] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 775.296845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 775.352818] alg: No test for adler32 (adler32-zlib) [ 776.355869] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 776.580592] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 778.247183] Key type lgssc registered [ 779.348657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 787.265024] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 787.316926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.005437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 793.782387] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 800.605662] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 808.172771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 808.549199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 814.469341] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 814.754658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:65) [ 820.498602] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 837.944170] LustreError: 26441:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 838.222435] Lustre: server umount lustre-OST0000 complete [ 852.466799] LustreError: 26642:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 852.472960] LustreError: 26642:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 852.703418] Lustre: server umount lustre-MDT0000 complete [ 864.113860] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 866.719733] Key type lgssc unregistered [ 866.973862] LNet: 27176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.993815] LNetError: 27176:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.025709] LNet: Removed LNI 192.168.201.121@tcp [ 867.856847] Key type .llcrypt unregistered [ 867.859394] Key type ._llcrypt unregistered [ 885.642743] Key type ._llcrypt registered [ 885.646421] Key type .llcrypt registered [ 885.780299] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 887.465161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 887.517602] alg: No test for adler32 (adler32-zlib) [ 888.572140] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 888.803309] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 890.545596] Key type lgssc registered [ 891.779393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 899.924914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 899.947034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 901.475880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.055219] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 912.349869] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 920.426723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 920.798841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 925.962584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 926.657228] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 932.193461] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 946.660154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 946.671628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.694049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 948.857178] LustreError: 29872:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 949.169597] Lustre: server umount lustre-OST0000 complete [ 953.073320] LustreError: 30073:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 953.080230] LustreError: 30073:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 953.341566] Lustre: server umount lustre-MDT0000 complete [ 964.317386] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 967.951712] Key type lgssc unregistered [ 968.263682] LNet: 30605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.269963] LNetError: 30605:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.319119] LNet: Removed LNI 192.168.201.121@tcp [ 970.410669] Key type .llcrypt unregistered [ 970.414062] Key type ._llcrypt unregistered [ 993.288408] Key type ._llcrypt registered [ 993.294154] Key type .llcrypt registered [ 993.446309] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 994.524221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 994.654411] alg: No test for adler32 (adler32-zlib) [ 995.689135] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 995.942949] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 997.623176] Key type lgssc registered [ 999.061481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1008.335128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1008.368357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.879214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.118859] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1019.733659] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1027.600225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1027.999748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1034.160657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 1034.736724] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1040.807254] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1049.044161] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 1049.574333] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.597745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1054.109929] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 1059.226633] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 1059.234268] Lustre: Skipped 1 previous similar message [ 1062.367136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1062.510748] LustreError: 33442:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1062.788477] Lustre: server umount lustre-MDT0000 complete [ 1075.124053] LustreError: 31977:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.21@tcp arrived at 1769755909 with bad export cookie 18117158163549437756 [ 1077.325927] LustreError: 31977:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755911 with bad export cookie 18117158163549437175 [ 1077.337175] LustreError: 31977:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1077.348434] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1083.467471] LustreError: 33670:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1083.473131] LustreError: 33670:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1083.616230] Lustre: server umount lustre-OST0000 complete [ 1097.677924] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1100.259158] Key type lgssc unregistered [ 1100.531791] LNet: 34356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.535535] LNetError: 34356:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1100.562650] LNet: Removed LNI 192.168.201.121@tcp [ 1101.291277] Key type .llcrypt unregistered [ 1101.294066] Key type ._llcrypt unregistered [ 1123.775691] Key type ._llcrypt registered [ 1123.777500] Key type .llcrypt registered [ 1123.898825] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1124.954669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1125.166757] alg: No test for adler32 (adler32-zlib) [ 1126.109887] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 1126.274350] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1127.911329] Key type lgssc registered [ 1128.885871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1136.133360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1136.156771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.879509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1143.060415] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1148.376087] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1154.894792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1155.227093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.422137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 1162.555139] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1169.388726] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1177.557027] Lustre: lustre-OST0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 1178.082879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.085899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1178.110151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1182.518350] LustreError: 37192:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1182.733673] Lustre: server umount lustre-OST0000 complete [ 1208.263811] LustreError: 37569:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1208.279841] LustreError: 37569:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1208.486719] Lustre: server umount lustre-MDT0000 complete [ 1220.495582] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1224.144739] Key type lgssc unregistered [ 1224.631692] LNet: 38104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.649306] LNetError: 38104:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.702114] LNet: Removed LNI 192.168.201.121@tcp [ 1226.951588] Key type .llcrypt unregistered [ 1226.954458] Key type ._llcrypt unregistered [ 1254.744779] Key type ._llcrypt registered [ 1254.748584] Key type .llcrypt registered [ 1254.815664] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1255.882559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1255.934069] alg: No test for adler32 (adler32-zlib) [ 1257.263710] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 1257.690429] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1259.500635] Key type lgssc registered [ 1260.720242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1271.693468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1271.724572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.146318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1276.797525] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1283.912154] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1292.284444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1292.908314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1298.034146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 1300.185400] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1306.389591] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1313.405805] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1317.636530] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1329.846482] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-01-30 1h56m02s [ 1331.421647] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1334.243057] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.267741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1335.341886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1335.348757] Lustre: Skipped 1 previous similar message [ 1339.627841] LustreError: 41265:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1339.819966] Lustre: server umount lustre-MDT0000 complete [ 1344.200880] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1351.878674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.996199] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1352.454146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.573101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:225) [ 1357.382993] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1361.517391] Lustre: DEBUG MARKER: Wait for df (30928) ... [ 1362.961905] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1363.281650] Lustre: DEBUG MARKER: done [ 1373.154243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1373.170111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1373.210112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1373.224782] Lustre: Skipped 1 previous similar message [ 1374.795315] LustreError: 42332:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1374.797918] LustreError: 42332:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1374.988517] Lustre: server umount lustre-OST0000 complete [ 1379.794652] LustreError: 42533:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1379.801304] LustreError: 42533:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1380.396906] Lustre: server umount lustre-MDT0000 complete [ 1394.795699] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1398.641731] Key type lgssc unregistered [ 1399.137349] LNet: 43066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1399.155045] LNetError: 43066:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1400.165872] LNet: Removed LNI 192.168.201.121@tcp [ 1401.148967] Key type .llcrypt unregistered [ 1401.155990] Key type ._llcrypt unregistered [ 1438.867772] Key type ._llcrypt registered [ 1438.870332] Key type .llcrypt registered [ 1438.973267] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1440.152623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1440.243629] alg: No test for adler32 (adler32-zlib) [ 1441.940934] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 1442.472445] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1444.207199] Key type lgssc registered [ 1446.131819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1457.169966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1457.197185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.547862] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1458.565973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1458.777947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.739541] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1470.021350] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1478.487711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1478.729150] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1478.927050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.379315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:257) [ 1485.919866] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1492.418380] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1500.350505] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-01-30 1h58m53s [ 1502.283577] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1510.485939] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1512.493972] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1512.687454] LustreError: 46886:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1532.901106] Lustre: lustre-MDT0000: Client 6d60f6a3-8278-47a4-b442-efbeeacf36b9 (at 192.168.201.21@tcp) reconnecting [ 1552.791284] LustreError: 46886:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1556.132768] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1559.468731] Lustre: DEBUG MARKER: done [ 1567.201680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1567.203712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1567.225782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1572.322613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1573.166125] LustreError: 47286:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1573.207670] Lustre: server umount lustre-OST0000 complete [ 1577.598911] LustreError: 47488:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1577.608484] LustreError: 47488:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1577.875269] Lustre: server umount lustre-MDT0000 complete [ 1590.329722] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1593.431419] Key type lgssc unregistered [ 1593.738633] LNet: 48021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.763432] LNetError: 48021:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.796978] LNet: Removed LNI 192.168.201.121@tcp [ 1594.757178] Key type .llcrypt unregistered [ 1594.760624] Key type ._llcrypt unregistered [ 1640.427697] Key type ._llcrypt registered [ 1640.429790] Key type .llcrypt registered [ 1640.553209] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1642.154825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1642.199700] alg: No test for adler32 (adler32-zlib) [ 1643.497948] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 1643.977797] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1645.673258] Key type lgssc registered [ 1646.875112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1650.543300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1650.563066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.010656] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1652.093463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1652.410426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1653.631839] Lustre: Failing over lustre-MDT0000 [ 1653.767787] LustreError: 50058:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1654.003352] Lustre: server umount lustre-MDT0000 complete [ 1661.896475] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:01:35 (1769756495) [ 1669.574766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.299653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.027981] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1683.206858] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1690.590667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1690.816462] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1691.094995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1698.420222] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1698.927563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:259 to 0x240000400:289) [ 1705.174393] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1712.565449] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1721.091288] LustreError: 52171:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1721.096585] LustreError: 52171:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1721.710298] Lustre: server umount lustre-MDT0000 complete [ 1726.028096] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1731.491515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1740.256105] Lustre: 49539:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769756558/real 1769756558] req@000000003c57984f x1855724168694144/t0(0) o400->MGC192.168.201.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769756574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1740.291192] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1740.309747] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.439036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.400452] Lustre: 49536:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769756564/real 1769756564] req@00000000510d688a x1855724168694528/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769756580 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1746.429222] Lustre: 49536:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1750.505110] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf59f7e0ffad3eec3 [ 1750.529510] Lustre: MGC192.168.201.121@tcp: Connection restored to 0@lo (at 0@lo) [ 1751.122914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1751.419634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1751.969056] Lustre: 49538:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769756569/real 1769756569] req@00000000610f8e33 x1855724168694784/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769756585 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1756.131038] Lustre: 49537:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769756574/real 1769756574] req@000000008eefa5ab x1855724168695040/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769756590 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1757.087682] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1764.920356] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1765.353968] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1772.130771] LustreError: 53641:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1772.145302] LustreError: 53641:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1772.288126] Lustre: server umount lustre-MDT0000 complete [ 1779.970649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1791.775859] Lustre: 49537:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769756609/real 1769756609] req@00000000aa1ebd19 x1855724168705536/t0(0) o400->MGC192.168.201.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769756625 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1791.828640] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1791.842576] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1792.150067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.183123] Lustre: 49536:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769756619/real 1769756619] req@00000000da4f5f84 x1855724168706176/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769756635 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1801.193448] LustreError: 49535:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a3cd7c2e x1855724168706944/t0(0) o250->MGC192.168.201.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1801.225710] Lustre: 49536:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1801.806313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.863591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1807.312515] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1813.790121] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1816.035850] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1821.644404] LustreError: 55103:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1821.649094] LustreError: 55103:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1821.863298] Lustre: server umount lustre-MDT0000 complete [ 1828.360686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1831.937537] Lustre: DEBUG MARKER: files compared the same [ 1833.842161] LustreError: 50606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756667 with bad export cookie 17699003667687142151 [ 1833.857432] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1843.211568] LustreError: 55698:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1843.219787] LustreError: 55698:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1843.357567] Lustre: server umount lustre-OST0000 complete [ 1858.288263] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1861.337815] Key type lgssc unregistered [ 1861.633723] LNet: 56383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1861.641043] LNetError: 56383:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1861.675062] LNet: Removed LNI 192.168.201.121@tcp [ 1862.742226] Key type .llcrypt unregistered [ 1862.745221] Key type ._llcrypt unregistered [ 1894.406705] Key type ._llcrypt registered [ 1894.408357] Key type .llcrypt registered [ 1894.534555] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1895.782142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1895.959691] alg: No test for adler32 (adler32-zlib) [ 1897.223600] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 1897.516128] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1899.271679] Key type lgssc registered [ 1900.944690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1911.015417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1911.045385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.791297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.262172] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug +malloc all [ 1925.849594] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1935.926811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1936.631332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1937.657158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:35 to 0x240000401:65) [ 1943.675443] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug +malloc all [ 1949.926531] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1963.497677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1963.521219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1966.997186] LustreError: 59228:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1967.146697] Lustre: server umount lustre-OST0000 complete [ 1972.148627] LustreError: 59429:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1972.168385] LustreError: 59429:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1972.871944] Lustre: server umount lustre-MDT0000 complete [ 1985.171977] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1989.039458] Key type lgssc unregistered [ 1989.439836] LNet: 59962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.450830] LNetError: 59962:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1989.488285] LNet: Removed LNI 192.168.201.121@tcp [ 1990.931099] Key type .llcrypt unregistered [ 1990.943579] Key type ._llcrypt unregistered [ 2018.812682] Key type ._llcrypt registered [ 2018.814194] Key type .llcrypt registered [ 2018.997887] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2020.567594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2020.885434] alg: No test for adler32 (adler32-zlib) [ 2022.063523] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 2022.350319] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 2024.016250] Key type lgssc registered [ 2025.449506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2035.545899] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2035.588241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2051.295346] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2056.479627] LustreError: 61334:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 2083.167325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.466405] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2097.250940] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2106.735936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.051229] Lustre: *** cfs_fail_loc=706, val=0*** [ 2107.662152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.825403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:35 to 0x240000401:97) [ 2113.445780] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2120.922902] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2127.331188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.350225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2129.998176] LustreError: 62860:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2130.144712] Lustre: server umount lustre-OST0000 complete [ 2135.629837] LustreError: 63064:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2135.660470] LustreError: 63064:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2136.234420] Lustre: server umount lustre-MDT0000 complete [ 2150.661717] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 2154.895859] Key type lgssc unregistered [ 2155.211628] LNet: 63598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.222538] LNetError: 63598:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.244991] LNet: Removed LNI 192.168.201.121@tcp [ 2156.424407] Key type .llcrypt unregistered [ 2156.428416] Key type ._llcrypt unregistered [ 2186.957170] Key type ._llcrypt registered [ 2186.961760] Key type .llcrypt registered [ 2187.145091] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2188.520420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2188.635665] alg: No test for adler32 (adler32-zlib) [ 2189.850873] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 2190.194581] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 2192.008121] Key type lgssc registered [ 2193.699258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2203.261725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2203.280507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.878572] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2218.929163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2219.493077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.226819] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2237.940071] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2237.972671] Lustre: MGS: Client 3f45c2b4-6fda-4310-bb00-5ca471a52ddc (at 0@lo) reconnecting [ 2238.670249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.793525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:35 to 0x240000401:129) [ 2243.820866] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2260.456559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.482596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2265.038180] LustreError: 66934:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 2265.148256] Lustre: server umount lustre-OST0000 complete [ 2269.362635] LustreError: 67136:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2269.370678] LustreError: 67136:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2269.457441] Lustre: server umount lustre-MDT0000 complete [ 2273.833258] LustreError: 67336:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2273.838444] LustreError: 67336:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2273.939846] Lustre: server umount MGS complete [ 2283.401521] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:11:56 (1769757116) [ 2312.806531] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 2325.341610] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2366.428308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.917577] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2384.306984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2384.657402] Lustre: lustre-OST0000: new disk, initializing [ 2384.663158] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2384.740239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.218996] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2400.408183] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2408.303249] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2408.315166] Lustre: MGS: Client 7d0fb7c7-ae17-4fdf-be75-57100f8444fa (at 0@lo) reconnecting [ 2408.510808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2408.522847] LustreError: 70410:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2408.543350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2408.614975] Lustre: lustre-MDT0000: new disk, initializing [ 2408.763879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2408.804475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2408.966577] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2410.023031] Lustre: 72064:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000002031b50a x1855724741219072/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1769757260 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2413.773083] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2419.121941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2419.136490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2419.173753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2441.194853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.223870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2446.306399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2446.738836] LustreError: 72583:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2446.741708] LustreError: 72583:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2446.956923] Lustre: server umount lustre-OST0000 complete [ 2457.597328] LustreError: 72785:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2457.604440] LustreError: 72785:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2457.768955] Lustre: server umount lustre-MDT0000 complete [ 2462.247532] Lustre: server umount MGS complete [ 2472.685649] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:15:05 (1769757305) [ 2488.763566] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 2492.155612] Key type lgssc unregistered [ 2492.407732] LNet: 74020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2492.413376] LNetError: 74020:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2492.423258] LNet: Removed LNI 192.168.201.121@tcp [ 2493.455176] Key type .llcrypt unregistered [ 2493.460690] Key type ._llcrypt unregistered [ 2513.443523] Key type ._llcrypt registered [ 2513.447209] Key type .llcrypt registered [ 2513.551487] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2514.491407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2514.595278] alg: No test for adler32 (adler32-zlib) [ 2515.897953] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 2516.344861] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 2518.103270] Key type lgssc registered [ 2519.617255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2528.509123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2528.536482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.993954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2535.139804] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2541.276407] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2548.641279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2549.088317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2555.254118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2555.867971] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2562.943907] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2575.859650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2575.862813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2575.877057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2578.456192] LustreError: 76670:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2578.686822] Lustre: server umount lustre-OST0000 complete [ 2583.762708] LustreError: 76871:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 2583.770484] LustreError: 76871:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2584.070849] Lustre: server umount lustre-MDT0000 complete [ 2597.062287] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 2600.817683] Key type lgssc unregistered [ 2601.245542] LNet: 77404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2601.254922] LNetError: 77404:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2601.268655] LNet: Removed LNI 192.168.201.121@tcp [ 2602.392772] Key type .llcrypt unregistered [ 2602.396736] Key type ._llcrypt unregistered [ 2620.376639] Key type ._llcrypt registered [ 2620.378959] Key type .llcrypt registered [ 2620.535211] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2622.027050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2622.145708] alg: No test for adler32 (adler32-zlib) [ 2623.253922] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 2623.489092] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 2625.136181] Key type lgssc registered [ 2626.894791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2633.553921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2633.571525] LustreError: 78436:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 2634.293865] LustreError: 78468:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 2634.298488] LustreError: 78436:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4281 [ 2634.301103] LustreError: 78436:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2634.307976] LustreError: 78436:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2634.330042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2636.185138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2641.996251] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing lsmod [ 2645.870321] LustreError: 78933:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 2645.978279] LustreError: 78944:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 2645.992708] LustreError: 78933:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4883 [ 2645.996042] LustreError: 78933:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2646.019675] LustreError: 78933:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2646.077640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2646.586370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.727778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 2651.244517] LustreError: 79137:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 2651.631205] Lustre: server umount lustre-MDT0000 complete [ 2656.977029] LustreError: 78485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757491 with bad export cookie 2051650672368002938 [ 2656.995540] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2667.525954] LustreError: 79338:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2667.534288] LustreError: 79338:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2667.675373] Lustre: server umount lustre-OST0000 complete [ 2678.822544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.695931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.035242] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2692.007475] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2700.458197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2700.717157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.826200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:97) [ 2707.800445] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2714.048758] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2725.351970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.378164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2727.033095] LustreError: 81262:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2727.045718] LustreError: 81262:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2727.254436] Lustre: server umount lustre-OST0000 complete [ 2732.073929] LustreError: 81463:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 2732.083937] LustreError: 81463:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2732.640877] Lustre: server umount lustre-MDT0000 complete [ 2745.699149] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 2749.140881] Key type lgssc unregistered [ 2749.497597] LNet: 81996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2749.506893] LNetError: 81996:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2749.525679] LNet: Removed LNI 192.168.201.121@tcp [ 2750.610929] Key type .llcrypt unregistered [ 2750.623421] Key type ._llcrypt unregistered [ 2778.571593] Key type ._llcrypt registered [ 2778.577508] Key type .llcrypt registered [ 2778.744646] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2780.232727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2780.308260] alg: No test for adler32 (adler32-zlib) [ 2781.669040] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 2782.030720] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 2783.783415] Key type lgssc registered [ 2785.179464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2794.831057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2794.852501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.508188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.197863] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2809.073976] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2817.096933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2817.495402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.628359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 2824.246417] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2830.619933] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2836.599748] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2839.975498] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2846.533024] LustreError: 84939:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2846.715879] Lustre: server umount lustre-OST0000 complete [ 2860.534559] LustreError: 85142:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 2860.545532] LustreError: 85142:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2860.781187] Lustre: server umount lustre-MDT0000 complete [ 2872.239133] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 2874.969715] Key type lgssc unregistered [ 2875.409560] LNet: 85675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2875.420642] LNetError: 85675:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2875.444212] LNet: Removed LNI 192.168.201.121@tcp [ 2876.418083] Key type .llcrypt unregistered [ 2876.419594] Key type ._llcrypt unregistered [ 2895.532522] Key type ._llcrypt registered [ 2895.536831] Key type .llcrypt registered [ 2895.698701] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2896.832921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2896.880332] alg: No test for adler32 (adler32-zlib) [ 2898.202994] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 2898.500700] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 2900.191261] Key type lgssc registered [ 2901.146796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2910.347315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2910.362599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.978908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2917.265905] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2923.592437] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2931.212318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2931.544757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.723794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 2937.962624] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2944.903095] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2958.306309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2958.311369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2958.318386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2960.457349] LustreError: 88322:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2960.733410] Lustre: server umount lustre-OST0000 complete [ 2964.881177] LustreError: 88524:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 2964.889820] LustreError: 88524:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2965.211788] Lustre: server umount lustre-MDT0000 complete [ 2977.424423] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 2981.081923] Key type lgssc unregistered [ 2981.408722] LNet: 89056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2981.419556] LNetError: 89056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2981.452903] LNet: Removed LNI 192.168.201.121@tcp [ 2982.500889] Key type .llcrypt unregistered [ 2982.503738] Key type ._llcrypt unregistered [ 3007.466557] Key type ._llcrypt registered [ 3007.468384] Key type .llcrypt registered [ 3007.625836] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3008.513038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3008.593215] alg: No test for adler32 (adler32-zlib) [ 3009.848801] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 3010.100314] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3011.807270] Key type lgssc registered [ 3012.949981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3021.965541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3022.000361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3023.627400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3028.287830] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3034.029842] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3040.949808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3041.469823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.943506] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3048.692989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 3053.932249] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3060.267229] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3064.030305] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3064.036408] Lustre: Skipped 1 previous similar message [ 3067.957427] Lustre: lustre-MDT0000: root_squash is set to 0:0 [ 3068.009357] Lustre: lustre-MDT0000: nosquash_nids is cleared [ 3068.514504] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3068.519442] Lustre: Skipped 1 previous similar message [ 3077.292099] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3083.081720] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.21@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3083.088608] Lustre: Skipped 1 previous similar message [ 3092.560604] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.201.21@tcp 192.168.0.[2,10]@tcp [ 3098.584122] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.21/24@tcp in log lustre-MDT0000 [ 3098.595654] Lustre: Skipped 1 previous similar message [ 3102.650061] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.21/24@tcp [ 3114.750760] LustreError: 93433:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3114.987852] Lustre: server umount lustre-OST0000 complete [ 3128.309654] LustreError: 93634:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3128.324916] LustreError: 93634:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3128.545868] Lustre: server umount lustre-MDT0000 complete [ 3139.509444] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3142.312681] Key type lgssc unregistered [ 3142.660784] LNet: 94168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3142.677030] LNetError: 94168:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3142.690592] LNet: Removed LNI 192.168.201.121@tcp [ 3143.468291] Key type .llcrypt unregistered [ 3143.470081] Key type ._llcrypt unregistered [ 3171.628610] Key type ._llcrypt registered [ 3171.630249] Key type .llcrypt registered [ 3171.745622] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3172.906254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3173.031781] alg: No test for adler32 (adler32-zlib) [ 3174.169529] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 3174.377254] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3176.089091] Key type lgssc registered [ 3177.359696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3184.969390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3185.010823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.623794] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3186.676486] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.21/24@tcp [ 3186.700422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.882847] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3197.857261] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3204.301213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3204.670908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3211.458182] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3211.837732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:201 to 0x240000400:225) [ 3217.327817] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3227.112893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3227.114017] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.168981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3232.193134] LustreError: 97213:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3232.349197] Lustre: server umount lustre-OST0000 complete [ 3236.758187] LustreError: 97415:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3236.764256] LustreError: 97415:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3237.007673] Lustre: server umount lustre-MDT0000 complete [ 3249.179714] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3252.462560] Key type lgssc unregistered [ 3252.848477] LNet: 97949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3252.874713] LNetError: 97949:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3252.894163] LNet: Removed LNI 192.168.201.121@tcp [ 3253.870630] Key type .llcrypt unregistered [ 3253.872519] Key type ._llcrypt unregistered [ 3278.513590] Key type ._llcrypt registered [ 3278.515471] Key type .llcrypt registered [ 3278.604145] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3291.573897] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3292.602561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3292.735702] alg: No test for adler32 (adler32-zlib) [ 3293.893370] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 3294.203202] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3295.879170] Key type lgssc registered [ 3297.159366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3324.667364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3324.698029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.992610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3326.026878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3326.099808] Lustre: lustre-MDT0000: new disk, initializing [ 3326.184185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.209083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3331.211492] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3338.738535] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3345.139092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3345.355443] Lustre: lustre-OST0000: new disk, initializing [ 3345.357709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3345.362444] Lustre: Skipped 1 previous similar message [ 3345.440764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.124357] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3355.249341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3355.258093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3355.294475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3357.990857] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3361.682677] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3361.828557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3365.353835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3365.364307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3365.376454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3370.472917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3375.594634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3378.145121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3378.205001] LustreError: 102697:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3378.317601] Lustre: server umount lustre-OST0000 complete [ 3382.269924] LustreError: 102900:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3382.274918] LustreError: 102900:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3382.417185] Lustre: server umount lustre-MDT0000 complete [ 3390.326351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.760162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3394.464487] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3399.341292] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3405.601236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3405.940671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3411.885946] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3417.186968] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3421.032056] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3421.267868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3430.024792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3430.190884] Lustre: lustre-OST0001: new disk, initializing [ 3430.196332] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3430.254531] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3435.617829] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3439.728710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3439.740533] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3439.769921] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3443.362965] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3443.587729] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3446.947143] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92c183410000.ost_server_uuid 50 [ 3448.334186] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92c183410000.ost_server_uuid in FULL state after 0 sec [ 3460.070690] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3460.076992] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.108869] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3461.498105] LustreError: 106311:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3461.502860] LustreError: 106311:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3461.592865] Lustre: server umount lustre-OST0001 complete [ 3465.186330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.198388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3470.318258] LustreError: 104176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.330201] LustreError: 104176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3470.712879] LustreError: 106512:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3470.719455] LustreError: 106512:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3470.817117] Lustre: server umount lustre-OST0000 complete [ 3473.826542] Lustre: server umount lustre-MDT0000 complete [ 3486.902887] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3489.587773] Key type lgssc unregistered [ 3489.834901] LNet: 107552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3489.842555] LNetError: 107552:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3490.920748] LNet: Removed LNI 192.168.201.121@tcp [ 3491.671081] Key type .llcrypt unregistered [ 3491.672970] Key type ._llcrypt unregistered [ 3519.914495] Key type ._llcrypt registered [ 3519.917097] Key type .llcrypt registered [ 3520.051772] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3533.238965] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3534.448812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3534.570498] alg: No test for adler32 (adler32-zlib) [ 3535.530468] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 3535.691219] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3537.351398] Key type lgssc registered [ 3538.567996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3563.870298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3563.900331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3565.134299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3565.164471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3565.222268] Lustre: lustre-MDT0000: new disk, initializing [ 3565.290828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3565.306986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3569.559703] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3577.071406] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3584.176192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3584.503582] Lustre: lustre-OST0000: new disk, initializing [ 3584.508513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3584.515904] Lustre: Skipped 1 previous similar message [ 3584.584474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.739469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3589.744935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3589.773810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3590.389949] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3597.064583] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3601.888788] Lustre: Failing over lustre-OST0000 [ 3601.942128] LustreError: 112607:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3602.082725] Lustre: server umount lustre-OST0000 complete [ 3619.515695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3619.897048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3619.916655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.933071] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3620.239248] Lustre: 109521:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769758438/real 1769758438] req@00000000b13d09cf x1855726151545472/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1769758454 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3620.293989] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3620.960122] Lustre: 109520:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769758439/real 1769758439] req@000000007ef861b1 x1855726151545984/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1769758455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3624.960510] Lustre: lustre-OST0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3624.965544] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3625.096350] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3626.017451] Lustre: 109521:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769758444/real 1769758444] req@00000000fac4ce17 x1855726151546368/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1769758460 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3626.042658] Lustre: 109521:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3628.552888] Lustre: Failing over lustre-MDT0000 [ 3628.669938] LustreError: 113580:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3628.674818] LustreError: 113580:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3628.827474] Lustre: server umount lustre-MDT0000 complete [ 3646.438277] Lustre: 109521:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769758464/real 1769758464] req@000000009e1e01f6 x1855726151552000/t0(0) o400->MGC192.168.201.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769758480 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3646.479771] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3646.499558] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3647.205646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3651.551326] Lustre: 109523:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769758469/real 1769758469] req@000000001480d6e3 x1855726151552384/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769758485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3651.582951] Lustre: 109523:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3656.678894] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf1fd32822f0ef690 [ 3656.693209] Lustre: MGC192.168.201.121@tcp: Connection restored to 0@lo (at 0@lo) [ 3657.078384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3657.119064] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3657.181165] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3657.216220] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3657.277446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3661.304661] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3662.112361] Lustre: 109523:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769758479/real 1769758479] req@00000000fada5da9 x1855726151552896/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769758495 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3662.152774] Lustre: 109523:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3671.529055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3671.559670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3674.168616] LustreError: 114482:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3674.177619] LustreError: 114482:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3674.272526] Lustre: server umount lustre-OST0000 complete [ 3677.934864] LustreError: 114683:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3677.939678] LustreError: 114683:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3678.163888] Lustre: server umount lustre-MDT0000 complete [ 3687.544743] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3689.701354] Key type lgssc unregistered [ 3689.923366] LNet: 115215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3689.932644] LNetError: 115215:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3689.952294] LNet: Removed LNI 192.168.201.121@tcp [ 3690.696635] Key type .llcrypt unregistered [ 3690.698805] Key type ._llcrypt unregistered [ 3721.462921] Key type ._llcrypt registered [ 3721.465640] Key type .llcrypt registered [ 3721.569902] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3722.840317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3722.888168] alg: No test for adler32 (adler32-zlib) [ 3723.938251] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 3724.161914] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3725.807186] Key type lgssc registered [ 3726.835362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3732.872992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3732.892297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3734.327737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.262240] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3744.625483] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3748.042497] LustreError: 117259:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3748.195746] Lustre: server umount lustre-MDT0000 complete [ 3757.911779] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3765.046155] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3786.343484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3786.535623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3786.564849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3786.627899] Lustre: lustre-MDT0000: new disk, initializing [ 3786.721683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3786.739845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3790.682094] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3797.556929] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3803.678415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3803.956866] Lustre: lustre-OST0000: new disk, initializing [ 3803.967326] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3803.979127] Lustre: Skipped 1 previous similar message [ 3804.040646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3804.878361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3804.896935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3804.941755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3809.542574] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3816.107898] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3820.448747] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3822.157129] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:37:35 (1769758655) [ 4030.433078] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4030.439269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4032.294860] LustreError: 122125:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4032.300048] LustreError: 122125:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4032.437634] Lustre: server umount lustre-MDT0000 complete [ 4035.301572] LustreError: 122000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758869 with bad export cookie 5884371007332020741 [ 4035.307308] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4035.312721] LustreError: 122000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4041.403279] LustreError: 122328:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4041.406037] LustreError: 122328:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4041.557819] Lustre: server umount lustre-OST0000 complete [ 4047.517571] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 4052.714818] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4071.370412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4071.519391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4071.540320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4071.599568] Lustre: lustre-MDT0000: new disk, initializing [ 4071.671928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4071.684575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4075.035299] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4080.886940] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4085.771608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4085.929464] Lustre: lustre-OST0000: new disk, initializing [ 4085.933226] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4085.940931] Lustre: Skipped 1 previous similar message [ 4086.008073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4089.969803] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4091.406346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4091.411790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4091.431843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4095.757749] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4099.003866] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4099.171675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4101.605806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4101.636206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4106.720198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4106.810305] LustreError: 126210:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4106.815435] LustreError: 126210:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4106.932779] Lustre: server umount lustre-OST0000 complete [ 4109.985245] Lustre: server umount lustre-MDT0000 complete [ 4115.423813] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:42:29 (1769758949) [ 4124.060526] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 4130.456737] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4151.882954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4152.094824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4152.113796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4152.177918] Lustre: lustre-MDT0000: new disk, initializing [ 4152.241654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.257455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4155.584766] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4161.920477] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4167.224506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4167.454517] Lustre: lustre-OST0000: new disk, initializing [ 4167.457739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4167.467929] Lustre: Skipped 1 previous similar message [ 4168.701027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4168.712353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4168.748380] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4172.443305] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4178.277523] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4184.033868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4184.039447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.046302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4185.150644] LustreError: 130645:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 4185.156402] LustreError: 130645:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4185.242633] Lustre: server umount lustre-OST0000 complete [ 4190.165936] Lustre: server umount lustre-MDT0000 complete [ 4196.550229] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:43:50 (1769759030) [ 4207.879853] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 4215.136931] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4236.294167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.486895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4236.510823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4236.576785] Lustre: lustre-MDT0000: new disk, initializing [ 4236.659859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4236.663973] Lustre: Skipped 1 previous similar message [ 4236.678698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4239.459271] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4244.422695] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4249.180806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4250.602814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4250.625971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4253.434925] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4258.949691] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4265.846527] LustreError: 135077:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 4265.850319] LustreError: 135077:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4265.999548] Lustre: server umount lustre-OST0000 complete [ 4278.368797] Lustre: server umount lustre-MDT0000 complete [ 4285.732240] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4287.632721] Key type lgssc unregistered [ 4287.824741] LNet: 135810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4287.830953] LNetError: 135810:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4287.850034] LNet: Removed LNI 192.168.201.121@tcp [ 4288.397745] Key type .llcrypt unregistered [ 4288.400281] Key type ._llcrypt unregistered [ 4303.523208] Key type ._llcrypt registered [ 4303.524755] Key type .llcrypt registered [ 4303.598627] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4304.241489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4304.254294] alg: No test for adler32 (adler32-zlib) [ 4305.276482] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 4305.436449] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4307.113270] Key type lgssc registered [ 4307.850512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4313.536849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4313.558944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.945795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4318.099565] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4321.975473] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4327.251453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4327.504883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4331.428362] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4335.051619] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4339.768458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 4360.161170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.172499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4362.680127] LustreError: 138650:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4362.768140] Lustre: server umount lustre-OST0000 complete [ 4365.214654] LustreError: 138851:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 4365.221867] LustreError: 138851:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4365.315267] Lustre: server umount lustre-MDT0000 complete [ 4372.323572] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4374.244697] Key type lgssc unregistered [ 4374.493767] LNet: 139382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4374.500434] LNetError: 139382:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4374.515777] LNet: Removed LNI 192.168.201.121@tcp [ 4375.052979] Key type .llcrypt unregistered [ 4375.055872] Key type ._llcrypt unregistered [ 4390.169294] Key type ._llcrypt registered [ 4390.170917] Key type .llcrypt registered [ 4390.243421] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4390.860647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4390.899339] alg: No test for adler32 (adler32-zlib) [ 4391.896594] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 4392.043160] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4393.663224] Key type lgssc registered [ 4394.521816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4399.565087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4399.578074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4400.911589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4403.432292] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4407.007619] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4411.771206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4412.015066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4415.806806] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4418.842387] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4425.317649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:65) [ 4430.817275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4430.819648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4430.829973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4431.793555] Lustre: lustre-OST0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 4435.713211] LustreError: 142200:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4435.766355] Lustre: server umount lustre-OST0000 complete [ 4439.484331] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff92c185c0e800.ost_server_uuid 50 [ 4440.583781] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4458.379940] LustreError: 142571:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 4458.382429] LustreError: 142571:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4458.467966] Lustre: server umount lustre-MDT0000 complete [ 4463.376536] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:48:17 (1769759297) [ 4467.251528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4467.586482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4470.001589] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4473.251377] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4477.241644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4477.503125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4478.567607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 4480.416975] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4483.214151] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4487.013910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4487.116414] Lustre: lustre-OST0001: new disk, initializing [ 4487.119350] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4487.156403] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4490.540654] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4494.534456] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4496.906411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4496.917326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4496.933138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 4496.940401] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 4496.960729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 4499.881342] Lustre: lustre-OST0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 4501.677069] LustreError: 145625:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4501.679104] LustreError: 145625:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4501.704286] Lustre: server umount lustre-OST0000 complete [ 4504.987057] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4505.139464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4507.108530] LustreError: 144042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4507.129808] LustreError: 144042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4510.107099] LustreError: 145538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4512.226177] LustreError: 145538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.344400] LustreError: 145538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.356733] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4517.364070] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.375109] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4522.106109] LustreError: 146081:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4522.108846] LustreError: 146081:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4522.219375] Lustre: server umount lustre-OST0001 complete [ 4524.817846] Lustre: server umount lustre-MDT0000 complete [ 4536.506383] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:49:30 (1769759370) [ 4540.539937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4540.696263] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4540.728152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4540.891249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4543.338464] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4546.895934] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4551.452768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4551.579793] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4555.489469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 4556.114302] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4559.968394] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4564.343956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4564.388236] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4564.498610] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4564.501133] Lustre: Skipped 1 previous similar message [ 4567.914684] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4570.849533] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4575.204489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4575.228616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4578.549550] LustreError: 149913:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 4578.553349] LustreError: 149913:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4578.588094] Lustre: server umount lustre-OST0000 complete [ 4580.323025] LustreError: 148365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.561166] LustreError: 148365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.574394] LustreError: 148365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4595.681686] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4595.689572] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4595.701208] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4595.704239] Lustre: Skipped 1 previous similar message [ 4597.357548] LustreError: 150138:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4597.361079] LustreError: 150138:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4597.413389] Lustre: server umount lustre-OST0001 complete [ 4599.856317] Lustre: server umount lustre-MDT0000 complete [ 4610.613609] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:50:44 (1769759444) [ 4617.249333] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 4621.921018] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4639.037210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4639.196319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4639.222237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4639.273405] Lustre: lustre-MDT0000: new disk, initializing [ 4639.349289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4639.358760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4642.093377] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4647.095924] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4650.830412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4650.979550] Lustre: lustre-OST0000: new disk, initializing [ 4650.981903] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4650.986823] Lustre: Skipped 1 previous similar message [ 4654.256971] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4655.055967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4655.062869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4655.090636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4659.655996] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4662.537443] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4662.691578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4665.313259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4665.313628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.320696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4678.111177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4678.137101] LustreError: 155321:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 4678.141472] LustreError: 155321:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4678.221372] Lustre: server umount lustre-OST0000 complete [ 4680.679232] Lustre: server umount lustre-MDT0000 complete [ 4685.979805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4688.785118] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4692.758316] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4697.018404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4701.007603] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4704.350229] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4706.979680] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4712.279397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 4717.025263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4717.028143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4719.291162] LustreError: 157725:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4719.295097] LustreError: 157725:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4719.355323] Lustre: server umount lustre-OST0000 complete [ 4722.374513] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4748.832182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4749.143900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4749.146868] Lustre: Skipped 3 previous similar messages [ 4752.988097] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4756.424570] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4757.346096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4757.356883] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4769.504634] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 11 sec [ 4772.835114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4772.844726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4772.856410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4772.862491] Lustre: Skipped 3 previous similar messages [ 4774.065837] Lustre: server umount lustre-OST0000 complete [ 4776.803425] Lustre: server umount lustre-MDT0000 complete [ 4782.281972] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:53:36 (1769759616) [ 4786.577340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.438259] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4793.009868] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4797.180566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.509153] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4805.555516] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4808.720834] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4808.850767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4813.734536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4813.836835] Lustre: lustre-OST0001: new disk, initializing [ 4813.839497] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4817.726790] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4822.628929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 4822.639534] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 4822.655164] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 4822.854261] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4825.856939] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4826.036614] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4827.616736] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4829.501251] LustreError: 162905:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4829.506286] LustreError: 162905:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4829.544874] Lustre: server umount lustre-OST0001 complete [ 4832.740596] LustreError: 160820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4832.753652] LustreError: 160820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4833.887084] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4834.027144] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4859.942150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4863.198359] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4865.515147] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4865.533809] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4866.984702] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4868.087868] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4870.624865] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4870.629236] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4870.640509] Lustre: Skipped 1 previous similar message [ 4875.403333] Lustre: server umount lustre-OST0001 complete [ 4875.682077] LustreError: 160822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4875.694412] LustreError: 160822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4901.344176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4901.346935] Lustre: Skipped 6 previous similar messages [ 4903.391069] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4903.490561] Lustre: server umount lustre-OST0000 complete [ 4911.808054] Lustre: server umount lustre-MDT0000 complete [ 4922.249156] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:55:56 (1769759756) [ 4925.667580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4925.820975] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4925.836381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4925.946950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4925.952251] Lustre: Skipped 4 previous similar messages [ 4928.203488] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4930.774167] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4933.724047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4933.814184] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4937.086381] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4939.868813] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4944.980712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4945.022360] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4948.013467] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4951.016393] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4951.942984] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4954.165699] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4954.286380] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4956.285373] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92c18478e000.ost_server_uuid 50 [ 4957.044399] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92c18478e000.ost_server_uuid in FULL state after 0 sec [ 4957.476739] Lustre: Permanently deactivating lustre-OST0001 [ 4957.482293] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4957.808611] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4958.500778] Lustre: Permanently reactivating lustre-OST0001 [ 4965.750171] LustreError: 168914:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4965.757573] LustreError: 168914:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4965.779725] Lustre: server umount lustre-OST0001 complete [ 4968.418159] LustreError: 166815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4968.427213] LustreError: 166815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4973.740627] Lustre: server umount lustre-OST0000 complete [ 4983.777949] LustreError: 168918:0:(osp_dev.c:692:osp_process_config()) lustre-OST0001-osc-MDT0000: unknown param osc.active=1 [ 4983.958706] Lustre: server umount lustre-MDT0000 complete [ 4992.772594] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:57:06 (1769759826) [ 4996.867950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4997.032222] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4997.061658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4997.067095] Lustre: Skipped 3 previous similar messages [ 4998.991483] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5001.392640] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5004.388426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5004.500175] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5004.516531] Lustre: Permanently deactivating lustre-OST0000 [ 5007.150800] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5009.589096] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5010.419955] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5010.423072] LustreError: 172073:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 5010.428410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 5013.132717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5013.172578] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5016.160283] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5018.913650] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5019.716560] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:65) [ 5019.869186] Lustre: Permanently reactivating lustre-OST0000 [ 5019.876846] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5019.883426] Lustre: Skipped 2 previous similar messages [ 5027.859032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5027.865606] Lustre: Skipped 3 previous similar messages [ 5027.868130] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5027.872532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.884458] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5027.886954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 5029.861511] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5029.866193] LustreError: Skipped 1 previous similar message [ 5034.771082] Lustre: server umount lustre-OST0001 complete [ 5050.847297] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5050.963552] Lustre: server umount lustre-OST0000 complete [ 5053.381583] Lustre: server umount lustre-MDT0000 complete [ 5058.719873] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5060.092325] Key type lgssc unregistered [ 5060.276148] LNet: 173999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5060.280158] LNetError: 173999:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5060.293468] LNet: Removed LNI 192.168.201.121@tcp [ 5060.737830] Key type .llcrypt unregistered [ 5060.740795] Key type ._llcrypt unregistered [ 5071.949868] Key type ._llcrypt registered [ 5071.952508] Key type .llcrypt registered [ 5072.014074] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 5079.339933] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5079.774467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5079.789340] alg: No test for adler32 (adler32-zlib) [ 5080.680150] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5080.819102] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5082.415833] Key type lgssc registered [ 5082.996205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5097.517500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5097.529287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5098.667653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5098.691441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5098.735847] Lustre: lustre-MDT0000: new disk, initializing [ 5098.806336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5098.819607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5100.862154] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5105.417860] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5109.444256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5109.608201] Lustre: lustre-OST0000: new disk, initializing [ 5109.616564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5109.622191] Lustre: Skipped 1 previous similar message [ 5109.670634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5112.695019] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5117.026125] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5118.479644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5118.486242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5118.505760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5125.514490] LustreError: 178446:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5131.166464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5131.291706] Lustre: lustre-OST0001: new disk, initializing [ 5131.294269] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5131.339359] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5134.640792] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5135.575194] LustreError: 178446:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5135.588149] LustreError: 178446:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5139.333459] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5140.007990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 5140.013349] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 5140.030126] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 5140.166468] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5145.599572] LustreError: 178446:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5147.050490] Lustre: lustre-OST0001: Not available for connect from 192.168.201.21@tcp (stopping) [ 5150.180229] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5150.185510] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.205226] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5152.155769] Lustre: lustre-OST0001: Not available for connect from 192.168.201.21@tcp (stopping) [ 5152.772158] LustreError: 179584:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5152.852980] Lustre: server umount lustre-OST0001 complete [ 5160.417845] LustreError: 177840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.431593] LustreError: 177840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5165.542332] LustreError: 177840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5169.631928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5169.636564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5169.644461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5170.658586] LustreError: 177839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.719566] LustreError: 179808:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 5172.722778] LustreError: 179808:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5172.795301] Lustre: server umount lustre-OST0000 complete [ 5180.815536] LustreError: 180009:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 5180.819776] LustreError: 180009:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5180.903465] Lustre: server umount lustre-MDT0000 complete [ 5185.861944] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5187.371450] Key type lgssc unregistered [ 5187.529772] LNet: 180542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5187.534969] LNetError: 180542:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5187.542500] LNet: Removed LNI 192.168.201.121@tcp [ 5187.931825] Key type .llcrypt unregistered [ 5187.933896] Key type ._llcrypt unregistered [ 5204.771143] Key type ._llcrypt registered [ 5204.772066] Key type .llcrypt registered [ 5204.830247] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5205.396784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5205.411842] alg: No test for adler32 (adler32-zlib) [ 5206.302202] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5206.428440] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5208.039131] Key type lgssc registered [ 5208.630555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5212.869440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5212.881909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.022497] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5214.035322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5214.123963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5216.217572] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5219.145881] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5222.728504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5222.845287] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5222.968569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5225.940919] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5228.009529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4 to 0x240000400:33) [ 5228.870090] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5274.081825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5274.085090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.090537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5277.035333] LustreError: 184060:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5277.094750] Lustre: server umount lustre-OST0000 complete [ 5279.299626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5285.429413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5285.630081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5288.197695] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5290.624485] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5294.817746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.826970] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5325.793436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5325.795093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5325.799972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5330.912693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5330.933094] LustreError: 185718:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5330.935498] LustreError: 185718:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5330.983342] Lustre: server umount lustre-OST0000 complete [ 5332.531542] LustreError: 185920:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 5332.534182] LustreError: 185920:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5332.591534] Lustre: server umount lustre-MDT0000 complete [ 5336.481946] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5337.808754] Key type lgssc unregistered [ 5337.947430] LNet: 186452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5337.949536] LNetError: 186452:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5337.958332] LNet: Removed LNI 192.168.201.121@tcp [ 5338.309632] Key type .llcrypt unregistered [ 5338.311040] Key type ._llcrypt unregistered [ 5348.098740] Key type ._llcrypt registered [ 5348.099850] Key type .llcrypt registered [ 5348.138526] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5348.483809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5348.493732] alg: No test for adler32 (adler32-zlib) [ 5349.370634] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5349.465052] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5351.055200] Key type lgssc registered [ 5351.484730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5354.358301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5354.365513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5355.547809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5356.987188] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5359.065110] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5361.551426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5361.672119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5363.717582] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5365.688364] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5365.796534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:42 to 0x240000400:65) [ 5375.968977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5375.971290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.977370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5381.088388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5381.236085] LustreError: 190037:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5381.297301] Lustre: server umount lustre-OST0000 complete [ 5383.005846] LustreError: 190241:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 5383.007916] LustreError: 190241:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5383.077835] Lustre: server umount lustre-MDT0000 complete [ 5387.127645] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5388.276844] Key type lgssc unregistered [ 5388.418452] LNet: 190775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5388.421558] LNetError: 190775:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5388.433357] LNet: Removed LNI 192.168.201.121@tcp [ 5388.753643] Key type .llcrypt unregistered [ 5388.755096] Key type ._llcrypt unregistered [ 5396.309770] Key type ._llcrypt registered [ 5396.311110] Key type .llcrypt registered [ 5396.354376] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5396.737685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5396.788304] alg: No test for adler32 (adler32-zlib) [ 5397.648491] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5397.742295] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5399.343262] Key type lgssc registered [ 5399.926944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5403.362488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5403.374682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.597328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.141879] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5408.420851] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5411.224077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5411.409152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5413.787827] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5414.501510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 5416.041469] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5425.119977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5425.120891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5425.128872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5427.183662] LustreError: 193538:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5427.223748] Lustre: server umount lustre-OST0000 complete [ 5428.942479] LustreError: 193740:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 5428.945089] LustreError: 193740:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5429.018853] Lustre: server umount lustre-MDT0000 complete [ 5433.130099] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5434.397822] Key type lgssc unregistered [ 5434.534503] LNet: 194271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5434.537943] LNetError: 194271:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5434.546352] LNet: Removed LNI 192.168.201.121@tcp [ 5434.900578] Key type .llcrypt unregistered [ 5434.902608] Key type ._llcrypt unregistered [ 5444.689379] Key type ._llcrypt registered [ 5444.690805] Key type .llcrypt registered [ 5444.735556] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5445.082696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5445.124718] alg: No test for adler32 (adler32-zlib) [ 5446.005033] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5446.108663] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5447.703216] Key type lgssc registered [ 5448.141100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5451.172238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5451.180268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5452.363834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.911377] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5456.099952] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5458.761528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5458.919186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5461.145897] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5461.991457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 5463.286526] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5476.335246] LustreError: 197857:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5476.374222] Lustre: server umount lustre-OST0000 complete [ 5487.085479] LustreError: 198059:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 5487.093071] LustreError: 198059:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5487.154065] Lustre: server umount lustre-MDT0000 complete [ 5491.638071] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5492.924658] Key type lgssc unregistered [ 5493.084457] LNet: 198591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5493.088196] LNetError: 198591:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5493.100648] LNet: Removed LNI 192.168.201.121@tcp [ 5493.495589] Key type .llcrypt unregistered [ 5493.497331] Key type ._llcrypt unregistered [ 5501.242823] Key type ._llcrypt registered [ 5501.244836] Key type .llcrypt registered [ 5501.301264] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5501.647516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5501.656810] alg: No test for adler32 (adler32-zlib) [ 5502.527261] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5502.657971] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5504.255176] Key type lgssc registered [ 5504.681624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5508.080606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5508.088290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5509.312822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.259261] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5515.244719] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5523.955586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5524.312886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.235358] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5533.922529] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5534.569030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 5550.048926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5550.061734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.068487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5552.054118] LustreError: 201396:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5552.120190] Lustre: server umount lustre-OST0000 complete [ 5553.887344] LustreError: 201597:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 5553.889643] LustreError: 201597:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5553.964445] Lustre: server umount lustre-MDT0000 complete [ 5558.871897] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5560.598501] Key type lgssc unregistered [ 5560.788174] LNet: 202131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5560.794809] LNetError: 202131:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5560.805433] LNet: Removed LNI 192.168.201.121@tcp [ 5561.244259] Key type .llcrypt unregistered [ 5561.246142] Key type ._llcrypt unregistered [ 5565.925829] Key type ._llcrypt registered [ 5565.927082] Key type .llcrypt registered [ 5565.972961] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5572.593435] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 5580.262417] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5580.783857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5580.798705] alg: No test for adler32 (adler32-zlib) [ 5581.678507] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5581.833718] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5583.472122] Key type lgssc registered [ 5583.978118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5596.525897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5596.535380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5597.655738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5597.672587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5597.714542] Lustre: lustre-MDT0000: new disk, initializing [ 5597.773927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5597.785034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5599.856203] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5604.373309] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5607.986479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5608.129890] Lustre: lustre-OST0000: new disk, initializing [ 5608.132964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5608.136986] Lustre: Skipped 1 previous similar message [ 5608.196722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5610.953697] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5612.567648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5612.571555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5612.585856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5614.835734] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5616.783306] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5616.915643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5622.753483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5622.757969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.763134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5623.931958] LustreError: 207008:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 5624.024691] Lustre: server umount lustre-OST0000 complete [ 5625.968154] LustreError: 207210:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5625.971141] LustreError: 207210:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5626.052111] Lustre: server umount lustre-MDT0000 complete [ 5629.613574] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:07:43 (1769760463) [ 5632.819206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5633.063323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5634.716775] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5637.093633] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5639.908488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5640.063045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5642.780392] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5644.955225] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5653.474083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5653.476400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5653.489561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5656.045444] LustreError: 209322:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5656.047853] LustreError: 209322:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5656.094541] Lustre: server umount lustre-OST0000 complete [ 5657.907725] Lustre: server umount lustre-MDT0000 complete [ 5661.988639] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5663.272498] Key type lgssc unregistered [ 5663.452500] LNet: 210056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5663.455939] LNetError: 210056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5663.466604] LNet: Removed LNI 192.168.201.121@tcp [ 5663.959167] Key type .llcrypt unregistered [ 5663.961099] Key type ._llcrypt unregistered [ 5673.466764] Key type ._llcrypt registered [ 5673.467682] Key type .llcrypt registered [ 5673.520277] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 5680.505285] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5680.952763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5681.007309] alg: No test for adler32 (adler32-zlib) [ 5681.897461] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5682.014661] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5683.632231] Key type lgssc registered [ 5684.111969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5700.188787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5700.202797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.325437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5701.349278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5701.396638] Lustre: lustre-MDT0000: new disk, initializing [ 5701.457843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5701.467288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5703.183382] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5706.742687] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5709.489552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5709.599361] Lustre: lustre-OST2710: new disk, initializing [ 5709.601849] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5709.604016] Lustre: Skipped 1 previous similar message [ 5709.640456] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5711.975329] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5715.504143] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5717.508280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:2710:ost [ 5717.511756] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:2710:ost] [ 5717.527896] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x240000400 [ 5718.129897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5718.195222] Lustre: lustre-OST03e8: new disk, initializing [ 5718.197105] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5718.224937] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5720.372670] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5722.413521] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5723.153358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:3e8:ost [ 5723.157161] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:3e8:ost] [ 5723.167611] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x280000400 [ 5725.251631] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5725.341549] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5726.955527] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5727.045728] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5728.224555] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.230995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5728.234114] Lustre: Skipped 1 previous similar message [ 5730.241526] LustreError: 216368:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5730.304650] Lustre: server umount lustre-MDT0000 complete [ 5731.870323] LustreError: 214291:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760565 with bad export cookie 16066117402383781828 [ 5731.872517] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5731.874892] LustreError: 214291:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5733.934099] LustreError: 216569:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 5733.936603] LustreError: 216569:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5733.953441] Lustre: server umount lustre-OST2710 complete [ 5735.425720] LustreError: 216769:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5735.428373] LustreError: 216769:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5735.471108] Lustre: server umount lustre-OST03e8 complete [ 5741.065409] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 5744.179797] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5754.489787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5754.588927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5754.600619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5754.638436] Lustre: lustre-MDT0000: new disk, initializing [ 5754.665880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5754.674281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5756.082095] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5759.303870] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5761.750082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5761.853229] Lustre: lustre-OST0000: new disk, initializing [ 5761.855299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5761.858702] Lustre: Skipped 1 previous similar message [ 5761.884673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5763.522384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5763.526285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5763.537230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5763.986809] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5767.344287] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5768.862948] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5768.942633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5773.792734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5773.793184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.800462] Lustre: Skipped 1 previous similar message [ 5773.802862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5775.723345] LustreError: 220951:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 5775.726466] LustreError: 220951:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5775.766238] Lustre: server umount lustre-OST0000 complete [ 5777.202126] Lustre: server umount lustre-MDT0000 complete [ 5780.167485] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:10:14 (1769760614) [ 5780.703788] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5781.341842] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:10:15 (1769760615) [ 5782.451806] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5791.008497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5791.107672] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5791.120524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5791.206523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.771600] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5795.021422] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5808.630237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5808.703763] Lustre: Denying initial registration attempt from nid 192.168.201.121@tcp, specified as failover [ 5808.707300] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5808.710622] LustreError: 223852:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 5808.713340] LustreError: 223852:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 5808.715634] LustreError: 223852:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5808.739789] Lustre: server umount lustre-OST0000 complete [ 5808.741855] LustreError: 223852:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5811.102528] LustreError: 224219:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5811.105112] LustreError: 224219:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5811.159336] Lustre: server umount lustre-MDT0000 complete [ 5815.352399] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5816.591927] Key type lgssc unregistered [ 5816.742911] LNet: 224751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5816.747522] LNetError: 224751:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5816.755299] LNet: Removed LNI 192.168.201.121@tcp [ 5817.169569] Key type .llcrypt unregistered [ 5817.171108] Key type ._llcrypt unregistered [ 5821.899019] Key type ._llcrypt registered [ 5821.900180] Key type .llcrypt registered [ 5821.946828] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5822.286759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5822.295172] alg: No test for adler32 (adler32-zlib) [ 5823.168913] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5823.283546] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5824.887231] Key type lgssc registered [ 5825.450529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5860.780806] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5864.062919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5864.071543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.180039] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5865.187534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5865.251047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5866.632398] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5868.561874] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5882.083973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5882.183640] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5882.266570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5884.513586] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5886.887841] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5887.860146] LustreError: 228719:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 5887.903225] Lustre: server umount lustre-OST0000 complete [ 5889.602828] LustreError: 228921:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5889.605734] LustreError: 228921:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5889.675323] Lustre: server umount lustre-MDT0000 complete [ 5894.107989] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5895.356064] Key type lgssc unregistered [ 5895.495338] LNet: 229451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5895.498396] LNetError: 229451:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5895.508316] LNet: Removed LNI 192.168.201.121@tcp [ 5895.861447] Key type .llcrypt unregistered [ 5895.862874] Key type ._llcrypt unregistered [ 5906.732503] Key type ._llcrypt registered [ 5906.734122] Key type .llcrypt registered [ 5906.773679] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5907.269326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5907.296821] alg: No test for adler32 (adler32-zlib) [ 5908.178300] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5908.269828] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5909.863176] Key type lgssc registered [ 5910.312349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5913.425658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5913.431809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5914.594274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5916.062549] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5917.982902] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5920.598033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5920.741816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5922.949437] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5925.082218] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5929.952276] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5929.958601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5931.424043] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 5934.855951] LustreError: 232268:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 5934.917586] Lustre: server umount lustre-MDT0000 complete [ 5937.535236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5940.708969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.761681] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5940.911072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5940.933940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:134 to 0x240000400:161) [ 5942.479943] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5944.578192] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5950.434270] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5953.353987] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 5955.552586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5955.557535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5960.671987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5960.757211] LustreError: 233728:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5960.760012] LustreError: 233728:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5960.796332] Lustre: server umount lustre-OST0000 complete [ 5962.292842] LustreError: 233930:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 5962.295582] LustreError: 233930:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5962.354605] Lustre: server umount lustre-MDT0000 complete [ 5966.311378] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5967.541605] Key type lgssc unregistered [ 5967.704973] LNet: 234461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5967.708409] LNetError: 234461:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5967.719293] LNet: Removed LNI 192.168.201.121@tcp [ 5968.045963] Key type .llcrypt unregistered [ 5968.047438] Key type ._llcrypt unregistered [ 5977.978669] Key type ._llcrypt registered [ 5977.979699] Key type .llcrypt registered [ 5978.022048] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5978.344883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5978.354930] alg: No test for adler32 (adler32-zlib) [ 5979.224859] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 5979.316351] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5980.903159] Key type lgssc registered [ 5981.325801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5984.081832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5984.089078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5985.246784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5986.636826] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5988.651036] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5989.986276] LustreError: 236428:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 5990.054838] Lustre: server umount lustre-MDT0000 complete [ 5996.194245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5996.335470] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5996.352854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5996.436540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5998.013035] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6000.179561] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6003.213156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6003.290338] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6003.294515] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6003.298976] LustreError: 238053:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 6003.302231] LustreError: 238053:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 6003.305669] LustreError: 238053:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6003.336707] Lustre: server umount lustre-OST0000 complete [ 6003.338940] LustreError: 238053:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6006.532621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6006.608529] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6006.672817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6008.958066] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6011.030404] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6011.392100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:134 to 0x240000400:193) [ 6014.028925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6014.062527] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6014.093600] Lustre: lustre-OST0001: new disk, initializing [ 6014.095790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6014.125233] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6016.503247] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6018.715181] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6019.782182] LustreError: 240144:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6019.784594] LustreError: 240144:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6019.801494] Lustre: server umount lustre-OST0001 complete [ 6021.118699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6021.124940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6026.208539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6026.209233] LustreError: 240352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.218545] LustreError: 240352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6027.179105] LustreError: 240344:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6027.181798] LustreError: 240344:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6027.216079] Lustre: server umount lustre-OST0000 complete [ 6028.728402] Lustre: server umount lustre-MDT0000 complete [ 6032.631985] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 6033.743291] Key type lgssc unregistered [ 6033.880460] LNet: 241085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6033.883959] LNetError: 241085:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6033.893410] LNet: Removed LNI 192.168.201.121@tcp [ 6034.205105] Key type .llcrypt unregistered [ 6034.206728] Key type ._llcrypt unregistered [ 6061.436023] Key type ._llcrypt registered [ 6061.437367] Key type .llcrypt registered [ 6061.485049] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 6067.774112] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 6068.204485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6068.216109] alg: No test for adler32 (adler32-zlib) [ 6069.074083] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6069.165739] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 6070.759153] Key type lgssc registered [ 6071.181919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6081.853084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6081.858666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6082.943269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6082.954777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6082.994198] Lustre: lustre-MDT0000: new disk, initializing [ 6083.021297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6083.028733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6084.381032] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6087.597588] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6090.028272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6090.132556] Lustre: lustre-OST0000: new disk, initializing [ 6090.135153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6090.138077] Lustre: Skipped 1 previous similar message [ 6090.162821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6091.713603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6091.716842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6091.725519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6092.146758] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6095.496687] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6097.026496] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6097.107723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6101.985565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6101.985626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6101.995241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6107.104317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6112.223116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6112.224108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6112.241074] LustreError: 246792:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6112.276837] Lustre: server umount lustre-OST0000 complete [ 6113.732501] LustreError: 246996:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6113.734850] LustreError: 246996:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6113.797567] Lustre: server umount lustre-MDT0000 complete [ 6116.821181] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:15:51 (1769760951) [ 6119.377438] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:15:53 (1769760953) [ 6122.320823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6122.488121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6123.870690] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6125.635911] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6127.791267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6127.891763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6129.809909] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6131.699819] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6137.311338] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6138.006641] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6138.602729] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6139.220517] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6139.994088] Lustre: Failing over lustre-MDT0000 [ 6140.158877] LustreError: 249643:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 6140.162063] LustreError: 249643:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6140.220483] Lustre: server umount lustre-MDT0000 complete [ 6153.556447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6153.608629] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6153.736817] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6153.756824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6153.783390] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6154.666478] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 6154.679698] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 6154.698730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:65) [ 6155.248472] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6158.621380] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6158.818791] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6158.858903] Lustre: 243461:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769760976/real 1769760976] req@00000000899b1f2f x1855728808646656/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769760992 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6159.274579] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6161.243036] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6161.991701] Lustre: Failing over lustre-MDT0000 [ 6162.086394] LustreError: 250732:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 6162.088925] LustreError: 250732:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6162.139256] Lustre: server umount lustre-MDT0000 complete [ 6163.039145] Lustre: 243463:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769760981/real 1769760981] req@000000009f5a0ac2 x1855728808646912/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769760997 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6168.159204] Lustre: 243461:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769760986/real 1769760986] req@00000000dbf38c66 x1855728808647168/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769761002 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6175.277395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.326802] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6175.439701] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6175.462573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6175.483399] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6176.899151] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6178.934865] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 6178.947605] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 6178.964748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:97) [ 6180.247804] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6180.575100] Lustre: 243463:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769760998/real 1769760998] req@0000000096f2d4b0 x1855728808654592/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769761014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6180.837402] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6180.856724] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6185.247136] Lustre: 243461:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769761003/real 1769761003] req@000000005c06a9d3 x1855728808654848/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769761019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6185.952190] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.956727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6189.308329] LustreError: 251769:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 6189.310450] LustreError: 251769:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6189.357514] Lustre: server umount lustre-MDT0000 complete [ 6190.642268] LustreError: 249412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761024 with bad export cookie 10215643400245374810 [ 6190.644354] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6190.648634] LustreError: 249412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6190.708334] Lustre: server umount lustre-OST0000 complete [ 6195.453878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6195.630511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6196.989201] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6198.801339] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6201.193050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6203.251108] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6205.144532] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6205.411633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:129) [ 6215.648646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.648708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6215.654409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6217.579065] LustreError: 254142:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6217.581611] LustreError: 254142:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6217.614037] Lustre: server umount lustre-OST0000 complete [ 6218.940487] Lustre: server umount lustre-MDT0000 complete [ 6222.503159] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 6223.565175] Key type lgssc unregistered [ 6223.696418] LNet: 254873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6223.699113] LNetError: 254873:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6223.709305] LNet: Removed LNI 192.168.201.121@tcp [ 6224.016390] Key type .llcrypt unregistered [ 6224.017756] Key type ._llcrypt unregistered [ 6233.293907] Key type ._llcrypt registered [ 6233.295094] Key type .llcrypt registered [ 6233.332459] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 6233.667637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6233.689068] alg: No test for adler32 (adler32-zlib) [ 6234.553818] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6234.638808] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 6236.223163] Key type lgssc registered [ 6236.614185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6239.236294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6239.242169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6240.418113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6241.795948] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6243.531420] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6245.827168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6245.940934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6247.843316] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6249.633932] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6250.019868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:161) [ 6260.193994] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6260.199221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6265.312164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6265.784962] LustreError: 257786:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 6265.833318] Lustre: server umount lustre-MDT0000 complete [ 6267.041166] LustreError: 257789:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761101 with bad export cookie 17759236223834266805 [ 6267.042623] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6267.045440] LustreError: 257789:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6267.062183] LustreError: 257990:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6267.065319] LustreError: 257990:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6267.108430] Lustre: server umount lustre-OST0000 complete [ 6271.974851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6272.188915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6273.645909] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6275.660893] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6277.908689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6278.027253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6279.905354] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6281.640542] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6282.150527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 6297.568020] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.572057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.895813] LustreError: 260263:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 6299.897457] LustreError: 260263:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6299.939099] Lustre: server umount lustre-MDT0000 complete [ 6301.193252] LustreError: 260177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761135 with bad export cookie 17759236223834267596 [ 6301.195181] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6301.197511] LustreError: 260177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6301.252327] Lustre: server umount lustre-OST0000 complete [ 6305.744702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6305.907912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6307.191547] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6308.912755] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6311.148784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6313.044359] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6314.851217] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6315.364867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:225) [ 6325.728720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6325.728904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6325.734283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6327.086077] LustreError: 262633:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6327.088323] LustreError: 262633:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6327.129288] Lustre: server umount lustre-OST0000 complete [ 6328.608881] Lustre: server umount lustre-MDT0000 complete [ 6332.309624] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 6333.407474] Key type lgssc unregistered [ 6333.545574] LNet: 263366:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6333.549348] LNetError: 263366:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6333.558483] LNet: Removed LNI 192.168.201.121@tcp [ 6333.867256] Key type .llcrypt unregistered [ 6333.868503] Key type ._llcrypt unregistered [ 6343.717317] Key type ._llcrypt registered [ 6343.718921] Key type .llcrypt registered [ 6343.766948] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 6344.227397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6344.233528] alg: No test for adler32 (adler32-zlib) [ 6345.101497] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6345.187490] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 6346.775158] Key type lgssc registered [ 6347.221184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6350.012734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6350.019262] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.022832] LustreError: 264760:0:(osd_handler.c:8564:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6350.032135] LustreError: 264760:0:(obd_config.c:859:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6350.035819] LustreError: 264760:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6350.038127] LustreError: 264760:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6350.041166] LustreError: 264760:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6352.637689] LDISKFS-fs (dm-1): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6352.640813] LustreError: 265156:0:(osd_handler.c:8564:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6352.651350] LustreError: 265156:0:(obd_config.c:859:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6352.658119] LustreError: 265156:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6352.665142] LustreError: 265156:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6352.672863] LustreError: 265156:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6358.153894] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 6359.228173] Key type lgssc unregistered [ 6359.371506] LNet: 265988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6359.374799] LNetError: 265988:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6359.384370] LNet: Removed LNI 192.168.201.121@tcp [ 6359.698825] Key type .llcrypt unregistered [ 6359.700110] Key type ._llcrypt unregistered [ 6364.983922] Key type ._llcrypt registered [ 6364.985109] Key type .llcrypt registered [ 6365.026414] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 6370.992632] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 6371.455102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6371.465871] alg: No test for adler32 (adler32-zlib) [ 6372.314205] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6372.397906] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 6373.983159] Key type lgssc registered [ 6374.331738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6383.247056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6383.255045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.350826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6384.362764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6384.400230] Lustre: lustre-MDT0000: new disk, initializing [ 6384.424135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6384.432274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6385.828121] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6388.925133] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6391.215484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6391.308863] Lustre: lustre-OST0000: new disk, initializing [ 6391.311217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6391.312810] Lustre: Skipped 1 previous similar message [ 6391.334443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6392.262157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6392.265738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6392.276115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6393.184189] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6396.226092] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6397.611088] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6397.684305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6402.528873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6402.528961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.535191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6404.397170] LustreError: 270490:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6404.432245] Lustre: server umount lustre-OST0000 complete [ 6405.718566] LustreError: 270693:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6405.721048] LustreError: 270693:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6405.773936] Lustre: server umount lustre-MDT0000 complete [ 6408.374972] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6408.886901] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:20:43 (1769761243) [ 6410.965200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.124879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6412.364891] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6414.060476] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6416.130611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6416.230976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6417.933083] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6419.581642] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6421.726642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6421.778457] Lustre: lustre-OST0001: new disk, initializing [ 6421.780860] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6421.804842] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6423.596923] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6426.695090] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6427.139189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 6427.143875] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 6427.152151] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 6432.225497] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6432.225541] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6432.229804] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6434.538052] LustreError: 273794:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6434.540144] LustreError: 273794:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6434.555571] Lustre: server umount lustre-OST0001 complete [ 6442.465319] LustreError: 272207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6442.469950] LustreError: 272207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6447.071834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6447.074207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6447.077974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6447.079528] Lustre: Skipped 1 previous similar message [ 6447.584508] LustreError: 272207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6452.524093] LustreError: 274017:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6452.526932] LustreError: 274017:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6452.564070] Lustre: server umount lustre-OST0000 complete [ 6460.032406] LustreError: 274218:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 6460.034052] LustreError: 274218:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6460.088864] Lustre: server umount lustre-MDT0000 complete [ 6463.527394] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 6464.627960] Key type lgssc unregistered [ 6464.759320] LNet: 274750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6464.762748] LNetError: 274750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6464.769300] LNet: Removed LNI 192.168.201.121@tcp [ 6465.066719] Key type .llcrypt unregistered [ 6465.067647] Key type ._llcrypt unregistered [ 6480.234180] Key type ._llcrypt registered [ 6480.235564] Key type .llcrypt registered [ 6480.271362] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 6480.543975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6480.555503] alg: No test for adler32 (adler32-zlib) [ 6481.403552] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6481.479745] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 6483.055159] Key type lgssc registered [ 6483.440097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6486.180426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6486.187937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6487.290431] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6487.297637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6487.358148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6488.726673] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6490.442325] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6491.083936] LustreError: 277711:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6491.132469] Lustre: server umount lustre-MDT0000 complete [ 6493.865570] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:22:08 (1769761328) [ 6495.932380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.092881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6497.264747] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6498.922772] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6501.021232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6501.082768] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6501.135921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6502.970259] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6504.615547] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6513.419274] Lustre: Permanently deactivating lustre-OST0000 [ 6513.424051] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6513.703719] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6514.001152] LustreError: 280080:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6514.003384] LustreError: 280080:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 6514.004902] LustreError: 280080:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6514.594115] LustreError: 280128:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 6514.597155] LustreError: 280128:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6521.322972] LustreError: 280225:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6521.325656] LustreError: 280225:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6521.357802] Lustre: server umount lustre-OST0000 complete [ 6522.230483] LustreError: 280377:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 6522.232726] LustreError: 280377:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6532.066896] LustreError: 280474:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6532.069475] LustreError: 280474:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6532.118530] Lustre: server umount lustre-MDT0000 complete [ 6534.822141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6536.105687] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6536.976342] LustreError: 281316:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6539.022385] Lustre: 281654:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.121@tcp [ 6539.684626] LustreError: 281751:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6539.687074] LustreError: 281751:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6539.710486] Lustre: server umount MGS complete [ 6542.557539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6542.713158] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6542.733906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6544.053464] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6545.930774] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6546.577851] Lustre: Permanently reactivating lustre-OST0000 [ 6546.582493] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6546.584473] Lustre: Skipped 1 previous similar message [ 6555.442701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6555.550106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6557.294315] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6559.024664] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6573.024474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6573.024477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.025021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6576.556075] LustreError: 284080:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6576.557781] LustreError: 284080:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6576.594888] Lustre: server umount lustre-OST0000 complete [ 6577.865817] Lustre: server umount lustre-MDT0000 complete [ 6581.082320] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 6582.000905] Key type lgssc unregistered [ 6582.114354] LNet: 284814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6582.116820] LNetError: 284814:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6582.122305] LNet: Removed LNI 192.168.201.121@tcp [ 6582.406324] Key type .llcrypt unregistered [ 6582.407266] Key type ._llcrypt unregistered [ 6587.044674] Key type ._llcrypt registered [ 6587.046315] Key type .llcrypt registered [ 6587.083088] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 6592.638373] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 6592.945510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6592.985077] alg: No test for adler32 (adler32-zlib) [ 6593.823699] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6593.896901] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 6595.471142] Key type lgssc registered [ 6595.790341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6604.090909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6604.101088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.185962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6605.196798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6605.236702] Lustre: lustre-MDT0000: new disk, initializing [ 6605.259327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6605.265091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6606.515602] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6609.492304] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6611.560037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6611.655348] Lustre: lustre-OST0000: new disk, initializing [ 6611.657098] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6611.659350] Lustre: Skipped 1 previous similar message [ 6611.678694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6613.454099] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6616.384974] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6617.645037] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6621.692941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6621.696626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6621.703549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6621.785240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 6626.784438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6626.785190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6626.790795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6628.461239] LustreError: 289341:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6628.495296] Lustre: server umount lustre-OST0000 complete [ 6629.713324] LustreError: 289543:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6629.714992] LustreError: 289543:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6629.767056] Lustre: server umount lustre-MDT0000 complete [ 6632.399914] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6632.925887] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:24:27 (1769761467) [ 6635.156217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6635.318168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6636.578892] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6638.276307] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6640.456806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6640.558664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6642.488480] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6644.260041] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6644.889271] Lustre: ctl-lustre-MDT0000: [0x0000000280000400-0x00000002c0000400]:0:mdt sequences allocated: rc = 0 [ 6651.840144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:mdt [ 6655.968755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6655.969034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6655.976995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6659.118081] LustreError: 291895:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6659.120457] LustreError: 291895:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6659.159082] Lustre: server umount lustre-OST0000 complete [ 6660.543894] Lustre: server umount lustre-MDT0000 complete [ 6664.125215] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 6665.197090] Key type lgssc unregistered [ 6665.337542] LNet: 292631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6665.341219] LNetError: 292631:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6665.350321] LNet: Removed LNI 192.168.201.121@tcp [ 6665.662784] Key type .llcrypt unregistered [ 6665.664323] Key type ._llcrypt unregistered [ 6682.736798] Key type ._llcrypt registered [ 6682.737880] Key type .llcrypt registered [ 6682.774149] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 6682.871305] Key type .llcrypt unregistered [ 6682.872397] Key type ._llcrypt unregistered [ 6689.137069] Key type ._llcrypt registered [ 6689.138729] Key type .llcrypt registered [ 6689.178645] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 6689.517524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6689.547051] alg: No test for adler32 (adler32-zlib) [ 6690.410330] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 6690.504667] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 6692.095156] Key type lgssc registered [ 6692.484794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6699.784241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6699.794882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6700.894712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6700.908688] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6700.947200] Lustre: lustre-MDT0000: new disk, initializing [ 6700.979924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6700.987214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6702.517647] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6705.841407] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6708.173662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6708.274809] Lustre: lustre-OST0000: new disk, initializing [ 6708.276847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6708.279464] Lustre: Skipped 1 previous similar message [ 6708.304902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6710.193488] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6710.459446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6710.462517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6710.471983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6713.291284] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6715.871828] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6715.876089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6720.991919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6721.593524] LustreError: 297626:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6721.639081] Lustre: server umount lustre-MDT0000 complete [ 6722.829129] LustreError: 296046:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761556 with bad export cookie 10273953628955961835 [ 6722.830684] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6722.832494] LustreError: 296046:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6722.849929] LustreError: 297828:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6722.851987] LustreError: 297828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6722.889430] Lustre: server umount lustre-OST0000 complete [ 6726.163464] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:26:00 (1769761560) [ 6728.654277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6728.837111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6730.105649] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6732.013248] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6745.542224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6745.609255] Lustre: Found index 0 for lustre-OST0000, updating log [ 6745.663460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6747.525524] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6749.219718] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6753.893533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5 to 0x240000400:33) [ 6759.392663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6759.392733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6759.397469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6761.708981] LustreError: 300098:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6761.711332] LustreError: 300098:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6761.746639] Lustre: server umount lustre-OST0000 complete [ 6774.325287] LustreError: 300353:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 6774.327436] LustreError: 300353:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6774.378121] Lustre: server umount lustre-MDT0000 complete [ 6777.112239] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:26:51 (1769761611) [ 6813.407269] LNet: 1 local NIs in recovery (showing 1): 192.168.201.121@tcp [ 7356.071557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7356.243055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.480755] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7359.143397] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7372.280610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7372.376090] Lustre: Found index 0 for lustre-OST0000, updating log [ 7372.458048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7374.478241] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7376.291291] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7376.987115] LustreError: 305224:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7376.990375] LustreError: 305224:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7377.024925] Lustre: server umount lustre-OST0000 complete [ 7378.418539] Lustre: server umount lustre-MDT0000 complete [ 7394.298988] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 7396.950448] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7405.753605] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:37:20 (1769762240) [ 7410.329093] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7411.268291] Key type lgssc unregistered [ 7415.519180] LNet: 1 local NIs in recovery (showing 1): 192.168.201.121@tcp [ 7434.975948] LNet: 308405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7434.978724] LNetError: 308405:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7501.541327] LNet: Removed LNI 192.168.201.121@tcp [ 7501.841406] Key type .llcrypt unregistered [ 7501.842679] Key type ._llcrypt unregistered [ 7507.948856] Key type ._llcrypt registered [ 7507.950056] Key type .llcrypt registered [ 7507.988741] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7508.332132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7508.374112] alg: No test for adler32 (adler32-zlib) [ 7509.218053] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 7509.297057] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 7510.871130] Key type lgssc registered [ 7511.203589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7512.578393] LNet: Added LNI 192.168.201.121@tcp100 [8/256/0/180] [ 7512.860311] LNet: Added LNI 192.168.201.121@tcp101 [8/256/0/180] [ 7513.143958] LNet: Added LNI 192.168.201.121@tcp102 [8/256/0/180] [ 7513.431529] LNet: Added LNI 192.168.201.121@tcp103 [8/256/0/180] [ 7513.712575] LNet: Added LNI 192.168.201.121@tcp104 [8/256/0/180] [ 7513.997982] LNet: Added LNI 192.168.201.121@tcp105 [8/256/0/180] [ 7514.311792] LNet: Added LNI 192.168.201.121@tcp106 [8/256/0/180] [ 7514.614111] LNet: Added LNI 192.168.201.121@tcp107 [8/256/0/180] [ 7514.936442] LNet: Added LNI 192.168.201.121@tcp108 [8/256/0/180] [ 7515.229173] LNet: Added LNI 192.168.201.121@tcp109 [8/256/0/180] [ 7515.537578] LNet: Added LNI 192.168.201.121@tcp110 [8/256/0/180] [ 7515.830451] LNet: Added LNI 192.168.201.121@tcp111 [8/256/0/180] [ 7516.122084] LNet: Added LNI 192.168.201.121@tcp112 [8/256/0/180] [ 7516.408144] LNet: Added LNI 192.168.201.121@tcp113 [8/256/0/180] [ 7516.687656] LNet: Added LNI 192.168.201.121@tcp114 [8/256/0/180] [ 7516.974624] LNet: Added LNI 192.168.201.121@tcp115 [8/256/0/180] [ 7517.258125] LNet: Added LNI 192.168.201.121@tcp116 [8/256/0/180] [ 7517.553817] LNet: Added LNI 192.168.201.121@tcp117 [8/256/0/180] [ 7517.860036] LNet: Added LNI 192.168.201.121@tcp118 [8/256/0/180] [ 7518.157458] LNet: Added LNI 192.168.201.121@tcp119 [8/256/0/180] [ 7518.467682] LNet: Added LNI 192.168.201.121@tcp120 [8/256/0/180] [ 7518.765340] LNet: Added LNI 192.168.201.121@tcp121 [8/256/0/180] [ 7519.073443] LNet: Added LNI 192.168.201.121@tcp122 [8/256/0/180] [ 7519.358645] LNet: Added LNI 192.168.201.121@tcp123 [8/256/0/180] [ 7519.655923] LNet: Added LNI 192.168.201.121@tcp124 [8/256/0/180] [ 7519.945809] LNet: Added LNI 192.168.201.121@tcp125 [8/256/0/180] [ 7520.236668] LNet: Added LNI 192.168.201.121@tcp126 [8/256/0/180] [ 7520.517997] LNet: Added LNI 192.168.201.121@tcp127 [8/256/0/180] [ 7520.799190] LNet: Added LNI 192.168.201.121@tcp128 [8/256/0/180] [ 7521.087569] LNet: Added LNI 192.168.201.121@tcp129 [8/256/0/180] [ 7521.373361] LNet: Added LNI 192.168.201.121@tcp130 [8/256/0/180] [ 7521.675365] LNet: Added LNI 192.168.201.121@tcp131 [8/256/0/180] [ 7521.972551] LNet: Added LNI 192.168.201.121@tcp132 [8/256/0/180] [ 7522.267668] LNet: Added LNI 192.168.201.121@tcp133 [8/256/0/180] [ 7522.553675] LNet: Added LNI 192.168.201.121@tcp134 [8/256/0/180] [ 7522.853029] LNet: Added LNI 192.168.201.121@tcp135 [8/256/0/180] [ 7525.154954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7525.160888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7526.211884] Lustre: 311382:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.121@tcp: can't add NID 192.168.201.121@tcp130: rc = -75 [ 7526.276720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7526.287439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7526.327390] Lustre: lustre-MDT0000: new disk, initializing [ 7526.357573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7526.363737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7527.679108] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7530.669731] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7532.719774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7532.812708] Lustre: lustre-OST0000: new disk, initializing [ 7532.814503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7532.816357] Lustre: Skipped 1 previous similar message [ 7532.841809] Lustre: 312293:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.121@tcp: can't add NID 192.168.201.121@tcp130: rc = -75 [ 7532.844323] Lustre: 312293:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7532.851395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7534.318743] Lustre: 312553:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.121@tcp: can't add NID 192.168.201.121@tcp130: rc = -75 [ 7534.320988] Lustre: 312553:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7534.343788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7534.346559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7534.356807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7534.625749] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7537.560399] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7544.800460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7544.802541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7544.806206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7549.920396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7554.015054] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7554.029478] LustreError: 313120:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 7554.057741] Lustre: server umount lustre-OST0000 complete [ 7555.251504] LustreError: 313322:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7555.253340] LustreError: 313322:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7555.312345] Lustre: server umount lustre-MDT0000 complete [ 7558.537114] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7559.477408] Key type lgssc unregistered [ 7559.619878] LNet: 313863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7559.622227] LNetError: 313863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7559.624031] LNet: Removed LNI 192.168.201.121@tcp [ 7560.671208] LNet: Removed LNI 192.168.201.121@tcp100 [ 7560.672599] LNetError: 313863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7560.674737] LNetError: 313863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7560.677146] LNet: Removed LNI 192.168.201.121@tcp101 [ 7561.695151] LNet: Removed LNI 192.168.201.121@tcp102 [ 7561.697029] LNetError: 313863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7561.699872] LNetError: 313863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7561.702886] LNet: Removed LNI 192.168.201.121@tcp103 [ 7562.719214] LNet: Removed LNI 192.168.201.121@tcp104 [ 7563.743121] LNet: Removed LNI 192.168.201.121@tcp105 [ 7563.744433] LNetError: 313863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7563.746845] LNetError: 313863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7563.749103] LNet: Removed LNI 192.168.201.121@tcp106 [ 7564.767290] LNet: Removed LNI 192.168.201.121@tcp107 [ 7565.791202] LNet: Removed LNI 192.168.201.121@tcp108 [ 7566.815150] LNet: Removed LNI 192.168.201.121@tcp109 [ 7567.839172] LNet: Removed LNI 192.168.201.121@tcp110 [ 7567.840983] LNetError: 313863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7567.843589] LNetError: 313863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7567.845578] LNet: Removed LNI 192.168.201.121@tcp111 [ 7568.863164] LNet: Removed LNI 192.168.201.121@tcp112 [ 7568.864980] LNet: Removed LNI 192.168.201.121@tcp113 [ 7568.866627] LNet: Removed LNI 192.168.201.121@tcp114 [ 7568.868300] LNet: Removed LNI 192.168.201.121@tcp115 [ 7568.869986] LNet: Removed LNI 192.168.201.121@tcp116 [ 7568.871697] LNet: Removed LNI 192.168.201.121@tcp117 [ 7568.873363] LNet: Removed LNI 192.168.201.121@tcp118 [ 7568.874924] LNet: Removed LNI 192.168.201.121@tcp119 [ 7568.876535] LNet: Removed LNI 192.168.201.121@tcp120 [ 7568.878196] LNet: Removed LNI 192.168.201.121@tcp121 [ 7568.879854] LNet: Removed LNI 192.168.201.121@tcp122 [ 7568.881541] LNet: Removed LNI 192.168.201.121@tcp123 [ 7568.883159] LNet: Removed LNI 192.168.201.121@tcp124 [ 7568.884786] LNet: Removed LNI 192.168.201.121@tcp125 [ 7568.886413] LNet: Removed LNI 192.168.201.121@tcp126 [ 7568.888081] LNet: Removed LNI 192.168.201.121@tcp127 [ 7568.889729] LNet: Removed LNI 192.168.201.121@tcp128 [ 7568.891391] LNet: Removed LNI 192.168.201.121@tcp129 [ 7568.893046] LNet: Removed LNI 192.168.201.121@tcp130 [ 7568.894781] LNet: Removed LNI 192.168.201.121@tcp131 [ 7568.896482] LNet: Removed LNI 192.168.201.121@tcp132 [ 7568.898169] LNet: Removed LNI 192.168.201.121@tcp133 [ 7568.899863] LNet: Removed LNI 192.168.201.121@tcp134 [ 7568.907417] LNet: Removed LNI 192.168.201.121@tcp135 [ 7569.159739] Key type .llcrypt unregistered [ 7569.161030] Key type ._llcrypt unregistered [ 7575.063472] Key type ._llcrypt registered [ 7575.064361] Key type .llcrypt registered [ 7575.096148] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 7580.807566] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7581.192493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7581.226781] alg: No test for adler32 (adler32-zlib) [ 7582.101470] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 7582.196151] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 7583.783389] Key type lgssc registered [ 7584.170944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7593.442137] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:40:27 (1769762427) [ 7598.449151] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7599.488028] Key type lgssc unregistered [ 7599.622328] LNet: 317297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7599.625484] LNetError: 317297:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7599.634334] LNet: Removed LNI 192.168.201.121@tcp [ 7599.931457] Key type .llcrypt unregistered [ 7599.932625] Key type ._llcrypt unregistered [ 7605.940846] Key type ._llcrypt registered [ 7605.941663] Key type .llcrypt registered [ 7605.974275] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7606.220629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7606.240064] alg: No test for adler32 (adler32-zlib) [ 7607.091966] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 7607.167065] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 7608.743147] Key type lgssc registered [ 7609.082230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7611.986384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7611.991261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7613.078887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7613.089487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7613.122521] Lustre: lustre-MDT0000: new disk, initializing [ 7613.145624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7613.151787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7614.347723] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7617.344993] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7619.448830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7619.533385] Lustre: lustre-OST0000: new disk, initializing [ 7619.535893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7619.538664] Lustre: Skipped 1 previous similar message [ 7619.556046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.739882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7619.743281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7619.752361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7621.268266] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7624.143603] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7625.223534] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing lnet_if_list [ 7626.625029] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7626.635033] LNet: Added LNI 192.168.201.121@tcp10 [8/256/0/180] [ 7628.223152] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7628.234483] LNet: Added LNI 192.168.201.121@tcp11 [8/256/0/180] [ 7629.407871] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7629.419836] LNet: Added LNI 192.168.201.121@tcp12 [8/256/0/180] [ 7630.650095] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7630.661730] LNet: Added LNI 192.168.201.121@tcp13 [8/256/0/180] [ 7631.969386] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7631.979189] LNet: Added LNI 192.168.201.121@tcp14 [8/256/0/180] [ 7633.211167] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7633.222488] LNet: Added LNI 192.168.201.121@tcp15 [8/256/0/180] [ 7634.458227] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7634.466886] LNet: Added LNI 192.168.201.121@tcp16 [8/256/0/180] [ 7635.754650] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7635.768399] LNet: Added LNI 192.168.201.121@tcp17 [8/256/0/180] [ 7637.229766] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7637.241442] LNet: Added LNI 192.168.201.121@tcp18 [8/256/0/180] [ 7638.856782] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7638.869377] LNet: Added LNI 192.168.201.121@tcp19 [8/256/0/180] [ 7640.415679] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7640.429824] LNet: Added LNI 192.168.201.121@tcp20 [8/256/0/180] [ 7641.934628] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7641.944541] LNet: Added LNI 192.168.201.121@tcp21 [8/256/0/180] [ 7643.386086] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7643.394247] LNet: Added LNI 192.168.201.121@tcp22 [8/256/0/180] [ 7644.840199] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7644.853288] LNet: Added LNI 192.168.201.121@tcp23 [8/256/0/180] [ 7646.281697] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7646.290405] LNet: Added LNI 192.168.201.121@tcp24 [8/256/0/180] [ 7647.984547] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7647.995884] LNet: Added LNI 192.168.201.121@tcp25 [8/256/0/180] [ 7649.752044] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7649.764407] LNet: Added LNI 192.168.201.121@tcp26 [8/256/0/180] [ 7651.577701] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7651.589041] LNet: Added LNI 192.168.201.121@tcp27 [8/256/0/180] [ 7653.442726] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7653.454303] LNet: Added LNI 192.168.201.121@tcp28 [8/256/0/180] [ 7655.347774] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7655.360749] LNet: Added LNI 192.168.201.121@tcp29 [8/256/0/180] [ 7657.289013] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7657.298451] LNet: Added LNI 192.168.201.121@tcp30 [8/256/0/180] [ 7659.364488] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7659.375950] LNet: Added LNI 192.168.201.121@tcp31 [8/256/0/180] [ 7661.469859] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7661.482672] LNet: Added LNI 192.168.201.121@tcp32 [8/256/0/180] [ 7663.626499] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7663.639476] LNet: Added LNI 192.168.201.121@tcp33 [8/256/0/180] [ 7665.829455] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7665.840826] LNet: Added LNI 192.168.201.121@tcp34 [8/256/0/180] [ 7667.839901] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7667.849737] LNet: Added LNI 192.168.201.121@tcp35 [8/256/0/180] [ 7669.744911] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7669.754881] LNet: Added LNI 192.168.201.121@tcp36 [8/256/0/180] [ 7671.762470] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7671.771567] LNet: Added LNI 192.168.201.121@tcp37 [8/256/0/180] [ 7673.720897] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7673.732577] LNet: Added LNI 192.168.201.121@tcp38 [8/256/0/180] [ 7675.673089] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7675.681931] LNet: Added LNI 192.168.201.121@tcp39 [8/256/0/180] [ 7677.670297] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7677.682505] LNet: Added LNI 192.168.201.121@tcp40 [8/256/0/180] [ 7680.091229] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7680.105898] LNet: Added LNI 192.168.201.121@tcp41 [8/256/0/180] [ 7682.447365] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7682.459254] LNet: Added LNI 192.168.201.121@tcp42 [8/256/0/180] [ 7684.921653] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7684.934863] LNet: Added LNI 192.168.201.121@tcp43 [8/256/0/180] [ 7687.450289] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7687.460510] LNet: Added LNI 192.168.201.121@tcp44 [8/256/0/180] [ 7689.963554] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7689.978784] LNet: Added LNI 192.168.201.121@tcp45 [8/256/0/180] [ 7692.468852] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7692.482208] LNet: Added LNI 192.168.201.121@tcp46 [8/256/0/180] [ 7695.344523] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7695.356835] LNet: Added LNI 192.168.201.121@tcp47 [8/256/0/180] [ 7697.964191] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7697.975262] LNet: Added LNI 192.168.201.121@tcp48 [8/256/0/180] [ 7700.746545] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7700.760587] LNet: Added LNI 192.168.201.121@tcp49 [8/256/0/180] [ 7703.583705] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7703.596315] LNet: Added LNI 192.168.201.121@tcp50 [8/256/0/180] [ 7712.224938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7712.228463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7716.859466] LustreError: 341605:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7716.907364] Lustre: server umount lustre-MDT0000 complete [ 7724.026546] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7726.788332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7726.823498] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7726.950688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7728.197859] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7729.857719] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7732.066633] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7737.312765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7737.313225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7737.316529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7742.432128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7742.829950] LustreError: 343301:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 7742.832401] LustreError: 343301:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7742.865923] Lustre: server umount lustre-OST0000 complete [ 7744.126415] LustreError: 343502:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 7744.128287] LustreError: 343502:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7744.188144] Lustre: server umount lustre-MDT0000 complete [ 7747.468845] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7748.466865] Key type lgssc unregistered [ 7748.615286] LNet: 344042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7748.617854] LNetError: 344042:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7748.619943] LNet: Removed LNI 192.168.201.121@tcp [ 7748.620972] LNet: Removed LNI 192.168.201.121@tcp10 [ 7748.622070] LNet: Removed LNI 192.168.201.121@tcp11 [ 7748.623093] LNet: Removed LNI 192.168.201.121@tcp12 [ 7748.624117] LNet: Removed LNI 192.168.201.121@tcp13 [ 7748.625125] LNet: Removed LNI 192.168.201.121@tcp14 [ 7748.626117] LNet: Removed LNI 192.168.201.121@tcp15 [ 7748.627136] LNet: Removed LNI 192.168.201.121@tcp16 [ 7748.628186] LNet: Removed LNI 192.168.201.121@tcp17 [ 7748.629270] LNet: Removed LNI 192.168.201.121@tcp18 [ 7748.630543] LNet: Removed LNI 192.168.201.121@tcp19 [ 7748.631691] LNet: Removed LNI 192.168.201.121@tcp20 [ 7748.632738] LNet: Removed LNI 192.168.201.121@tcp21 [ 7748.633822] LNet: Removed LNI 192.168.201.121@tcp22 [ 7748.634889] LNet: Removed LNI 192.168.201.121@tcp23 [ 7748.635912] LNet: Removed LNI 192.168.201.121@tcp24 [ 7748.637079] LNet: Removed LNI 192.168.201.121@tcp25 [ 7748.639144] LNet: Removed LNI 192.168.201.121@tcp26 [ 7748.640238] LNet: Removed LNI 192.168.201.121@tcp27 [ 7748.641237] LNet: Removed LNI 192.168.201.121@tcp28 [ 7748.642224] LNet: Removed LNI 192.168.201.121@tcp29 [ 7748.643241] LNet: Removed LNI 192.168.201.121@tcp30 [ 7748.644223] LNet: Removed LNI 192.168.201.121@tcp31 [ 7748.645321] LNet: Removed LNI 192.168.201.121@tcp32 [ 7748.646780] LNet: Removed LNI 192.168.201.121@tcp33 [ 7748.647873] LNet: Removed LNI 192.168.201.121@tcp34 [ 7748.648950] LNet: Removed LNI 192.168.201.121@tcp35 [ 7748.650040] LNet: Removed LNI 192.168.201.121@tcp36 [ 7748.651090] LNet: Removed LNI 192.168.201.121@tcp37 [ 7748.652154] LNet: Removed LNI 192.168.201.121@tcp38 [ 7748.653234] LNet: Removed LNI 192.168.201.121@tcp39 [ 7748.654319] LNet: Removed LNI 192.168.201.121@tcp40 [ 7748.655447] LNet: Removed LNI 192.168.201.121@tcp41 [ 7748.656466] LNet: Removed LNI 192.168.201.121@tcp42 [ 7748.657509] LNet: Removed LNI 192.168.201.121@tcp43 [ 7748.658550] LNet: Removed LNI 192.168.201.121@tcp44 [ 7748.659555] LNet: Removed LNI 192.168.201.121@tcp45 [ 7748.660523] LNet: Removed LNI 192.168.201.121@tcp46 [ 7748.661735] LNet: Removed LNI 192.168.201.121@tcp47 [ 7748.662971] LNet: Removed LNI 192.168.201.121@tcp48 [ 7748.664453] LNet: Removed LNI 192.168.201.121@tcp49 [ 7748.671550] LNet: Removed LNI 192.168.201.121@tcp50 [ 7748.920173] Key type .llcrypt unregistered [ 7748.921552] Key type ._llcrypt unregistered [ 7756.050109] Key type ._llcrypt registered [ 7756.051168] Key type .llcrypt registered [ 7756.087944] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7756.446641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7756.480109] alg: No test for adler32 (adler32-zlib) [ 7757.335118] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 7757.412522] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 7758.991102] Key type lgssc registered [ 7759.311972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7761.770445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7761.777688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7762.924846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7764.163409] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7765.871302] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7768.022882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7768.126676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7770.043178] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7771.902562] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7773.283255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:65) [ 7781.564582] loop0: detected capacity change from 0 to 204800000 [ 7781.678429] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7781.681370] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7781.685065] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7783.590434] loop0: detected capacity change from 0 to 204800000 [ 7783.707170] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7783.712608] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7783.717557] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7785.407324] loop0: detected capacity change from 0 to 204800000 [ 7787.375158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7787.424455] Lustre: fs15246-OST0000: new disk, initializing [ 7787.426642] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7787.449475] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7789.326464] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7793.053084] loop1: detected capacity change from 0 to 204800000 [ 7794.890317] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7794.933182] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7794.963395] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7795.010910] Lustre: fs15246-MDT0000: new disk, initializing [ 7795.063254] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7795.072623] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7795.116133] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7796.127715] Lustre: 348819:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000038fdfba2 x1855730578637440/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1769762646 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7796.421985] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7803.878790] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7805.215244] LustreError: 348819:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7805.218387] LustreError: 348819:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7805.221352] LustreError: 348819:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7808.994478] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7809.851163] LustreError: 349938:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7809.877525] Lustre: server umount fs15246-MDT0000 complete [ 7812.981095] LustreError: 350443:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 7812.984022] LustreError: 350443:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7813.000231] Lustre: server umount fs15246-OST0000 complete [ 7819.233289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7819.237207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7822.507574] LustreError: 350944:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7822.511053] LustreError: 350944:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7822.544651] Lustre: server umount lustre-OST0000 complete [ 7823.907026] Lustre: server umount lustre-MDT0000 complete [ 7827.323378] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7828.363648] Key type lgssc unregistered [ 7828.498452] LNet: 351677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7828.500691] LNetError: 351677:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7828.510328] LNet: Removed LNI 192.168.201.121@tcp [ 7828.808585] Key type .llcrypt unregistered [ 7828.810077] Key type ._llcrypt unregistered [ 7835.811245] Key type ._llcrypt registered [ 7835.812155] Key type .llcrypt registered [ 7835.847205] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7836.211764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7836.248265] alg: No test for adler32 (adler32-zlib) [ 7837.101892] Lustre: Lustre: Build Version: 2.17.50_86_g2f7fa4d [ 7837.184850] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 7838.767179] Key type lgssc registered [ 7839.123535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7849.994334] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 7852.686875] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7862.034528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7862.040026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7863.128917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7863.139371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7863.174360] Lustre: lustre-MDT0000: new disk, initializing [ 7863.196867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7863.203026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7864.450791] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7867.502910] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7869.707431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7869.799619] Lustre: lustre-OST0000: new disk, initializing [ 7869.801410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7869.802953] Lustre: Skipped 1 previous similar message [ 7869.826346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7871.621728] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7874.629458] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7876.035667] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7879.164142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7879.167079] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7879.175109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7880.175434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 7884.257900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7884.257994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7884.264702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7886.894092] LustreError: 357486:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 7886.930057] Lustre: server umount lustre-OST0000 complete [ 7888.160181] LustreError: 357688:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7888.163637] LustreError: 357688:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7888.218593] Lustre: server umount lustre-MDT0000 complete [ 7894.916633] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 7897.949430] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7905.703094] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7614 sec ========== 03:45:39 (1769762739) [ 7906.236225] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:45:40 (1769762740) === [ 7907.228442] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:45:41 (1769762741) === [ 7917.792451] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7918.865360] Key type lgssc unregistered [ 7919.012310] LNet: 361710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7919.015833] LNetError: 361710:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7919.024412] LNet: Removed LNI 192.168.201.121@tcp [ 7919.310280] Key type .llcrypt unregistered [ 7919.311182] Key type ._llcrypt unregistered