[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483604944 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001025] APIC: Switch to symmetric I/O mode setup [ 0.003240] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007915] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010138] LSM: Security Framework initializing [ 0.012028] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.022828] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027155] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030124] *** VALIDATE tmpfs *** [ 0.031471] *** VALIDATE proc *** [ 0.033233] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035290] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038036] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042387] debug: unmapping init [mem 0xffffffff91259000-0xffffffff91260fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045742] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.048873] ... value mask: 0000ffffffffffff [ 0.049019] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052300] rcu: Hierarchical SRCU implementation. [ 0.054474] smp: Bringing up secondary CPUs ... [ 0.055530] x86: Booting SMP configuration: [ 0.056027] .... node #0, CPUs: #1 #2 #3 [ 0.059645] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142000] node 0 deferred pages initialised in 78ms [ 0.146195] devtmpfs: initialized [ 0.147203] x86/mm: Memory block size: 128MB [ 0.149839] gcov: version magic: 0x41383552 [ 0.151217] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153243] pinctrl core: initialized pinctrl subsystem [ 0.154180] [ 0.154683] ************************************************************* [ 0.155017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156014] ** ** [ 0.157015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158016] ** ** [ 0.159013] ** This means that this kernel is built to expose internal ** [ 0.160011] ** IOMMU data structures, which may compromise security on ** [ 0.161013] ** your system. ** [ 0.162014] ** ** [ 0.163012] ** If you see this message and you are not debugging the ** [ 0.164012] ** kernel, report this immediately to your vendor! ** [ 0.165013] ** ** [ 0.166013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167011] ************************************************************* [ 0.168711] NET: Registered protocol family 16 [ 0.169432] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.172407] cpuidle: using governor menu [ 0.175042] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177487] PCI: Using configuration type 1 for base access [ 0.180126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.190161] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193127] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.196149] cryptd: max_cpu_qlen set to 1000 [ 0.199659] ACPI: Added _OSI(Module Device) [ 0.201017] ACPI: Added _OSI(Processor Device) [ 0.203014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.209150] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.216523] ACPI: Interpreter enabled [ 0.217051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.219013] ACPI: Using IOAPIC for interrupt routing [ 0.220092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223399] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232597] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.234041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.240093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.245420] acpiphp: Slot [2] registered [ 0.246118] acpiphp: Slot [5] registered [ 0.248106] acpiphp: Slot [6] registered [ 0.249135] acpiphp: Slot [7] registered [ 0.251132] acpiphp: Slot [8] registered [ 0.252119] acpiphp: Slot [9] registered [ 0.254125] acpiphp: Slot [10] registered [ 0.255112] acpiphp: Slot [3] registered [ 0.256096] acpiphp: Slot [4] registered [ 0.258106] acpiphp: Slot [11] registered [ 0.259127] acpiphp: Slot [12] registered [ 0.261128] acpiphp: Slot [13] registered [ 0.262129] acpiphp: Slot [14] registered [ 0.264113] acpiphp: Slot [15] registered [ 0.265116] acpiphp: Slot [16] registered [ 0.267098] acpiphp: Slot [17] registered [ 0.268137] acpiphp: Slot [18] registered [ 0.270097] acpiphp: Slot [19] registered [ 0.271106] acpiphp: Slot [20] registered [ 0.273094] acpiphp: Slot [21] registered [ 0.274147] acpiphp: Slot [22] registered [ 0.276093] acpiphp: Slot [23] registered [ 0.277167] acpiphp: Slot [24] registered [ 0.279111] acpiphp: Slot [25] registered [ 0.280106] acpiphp: Slot [26] registered [ 0.282108] acpiphp: Slot [27] registered [ 0.283103] acpiphp: Slot [28] registered [ 0.284094] acpiphp: Slot [29] registered [ 0.286103] acpiphp: Slot [30] registered [ 0.287127] acpiphp: Slot [31] registered [ 0.289078] PCI host bridge to bus 0000:00 [ 0.290019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.293029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.295028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.297021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.298017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.300020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.301447] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.305079] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.307937] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.315537] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.320056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.323020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.326032] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.328019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.330620] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.333811] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.338045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.340828] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.346019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.358015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.363013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.369443] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.378017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.396025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.421020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.434215] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.439016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.448020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.472018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.481544] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.488016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.495014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.508016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.520640] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.526016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.533017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.548025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.558080] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.567018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.572017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.584016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.594840] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.599015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.606019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.621018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.631620] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.634305] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.635315] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.637581] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.640235] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.645098] iommu: Default domain type: Passthrough [ 0.647639] SCSI subsystem initialized [ 0.649133] ACPI: bus type USB registered [ 0.650106] usbcore: registered new interface driver usbfs [ 0.652275] usbcore: registered new interface driver hub [ 0.654337] usbcore: registered new device driver usb [ 0.656466] pps_core: LinuxPPS API ver. 1 registered [ 0.658011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.661064] PTP clock support registered [ 0.663093] EDAC MC: Ver: 3.0.0 [ 0.665152] PCI: Using ACPI for IRQ routing [ 0.666507] NetLabel: Initializing [ 0.668013] NetLabel: domain hash size = 128 [ 0.669008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.671087] NetLabel: unlabeled traffic allowed by default [ 0.673138] vgaarb: loaded [ 0.675279] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.677015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.683018] clocksource: Switched to clocksource kvm-clock [ 0.790930] VFS: Disk quotas dquot_6.6.0 [ 0.792606] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.794799] *** VALIDATE ramfs *** [ 0.795595] *** VALIDATE hugetlbfs *** [ 0.796590] pnp: PnP ACPI init [ 0.798188] pnp: PnP ACPI: found 6 devices [ 0.815136] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.817923] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.819689] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.821741] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.824057] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.826421] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.829151] NET: Registered protocol family 2 [ 0.831343] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.835560] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.839148] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.846050] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.849521] TCP: Hash tables configured (established 65536 bind 65536) [ 0.852409] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.855444] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.858112] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.861017] NET: Registered protocol family 1 [ 0.863552] RPC: Registered named UNIX socket transport module. [ 0.866241] RPC: Registered udp transport module. [ 0.867933] RPC: Registered tcp transport module. [ 0.869635] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.871992] NET: Registered protocol family 44 [ 0.873682] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.875672] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.877486] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.879543] PCI: CLS 0 bytes, default 64 [ 0.881687] Unpacking initramfs... [ 2.307251] debug: unmapping init [mem 0xffff8d7afcc54000-0xffff8d7afffbffff] [ 2.310833] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.312829] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.315311] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.792798] Initialise system trusted keyrings [ 2.794454] Key type blacklist registered [ 2.796360] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.806447] zbud: loaded [ 2.810538] *** VALIDATE nfs *** [ 2.811745] *** VALIDATE nfs4 *** [ 2.813978] pstore: using deflate compression [ 2.817838] Platform Keyring initialized [ 2.928506] NET: Registered protocol family 38 [ 2.931504] Key type asymmetric registered [ 2.933009] Asymmetric key parser 'x509' registered [ 2.934769] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.938150] io scheduler mq-deadline registered [ 2.939875] io scheduler kyber registered [ 2.941477] io scheduler bfq registered [ 2.943134] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.945664] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.948397] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.951094] ACPI: Power Button [PWRF] [ 2.957054] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.963396] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.978188] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.986465] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.003705] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.036760] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.065413] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.069490] Non-volatile memory driver v1.3 [ 3.070729] Linux agpgart interface v0.103 [ 3.096607] virtio_blk virtio1: [vda] 133936 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.099111] vda: detected capacity change from 0 to 68575232 [ 3.114076] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.116681] vdb: detected capacity change from 0 to 1073741824 [ 3.132994] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.135730] vdc: detected capacity change from 0 to 2621440000 [ 3.150953] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.154118] vdd: detected capacity change from 0 to 2621440000 [ 3.169518] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.172724] vde: detected capacity change from 0 to 4294967296 [ 3.186503] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.188674] vdf: detected capacity change from 0 to 4294967296 [ 3.193635] libphy: Fixed MDIO Bus: probed [ 3.198054] usbcore: registered new interface driver usbserial_generic [ 3.200060] usbserial: USB Serial support registered for generic [ 3.202086] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.205635] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.207240] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.209165] mousedev: PS/2 mouse device common for all mice [ 3.211419] rtc_cmos 00:05: RTC can wake from S4 [ 3.213209] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.216072] rtc_cmos 00:05: registered as rtc0 [ 3.217894] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.220080] intel_pstate: CPU model not supported [ 3.221518] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.225505] hid: raw HID events driver (C) Jiri Kosina [ 3.226941] usbcore: registered new interface driver usbhid [ 3.227645] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.228146] usbhid: USB HID core driver [ 3.232144] drop_monitor: Initializing network drop monitor service [ 3.233596] Initializing XFRM netlink socket [ 3.235081] NET: Registered protocol family 10 [ 3.237830] Segment Routing with IPv6 [ 3.239261] NET: Registered protocol family 17 [ 3.241760] mpls_gso: MPLS GSO support [ 3.247211] RAS: Correctable Errors collector initialized. [ 3.248884] AVX version of gcm_enc/dec engaged. [ 3.250023] AES CTR mode by8 optimization enabled [ 3.313254] sched_clock: Marking stable (3313231913, 0)->(4220246717, -907014804) [ 3.315879] registered taskstats version 1 [ 3.317826] Loading compiled-in X.509 certificates [ 3.319744] zswap: loaded using pool lzo/zbud [ 3.341975] Key type big_key registered [ 3.353551] Key type encrypted registered [ 3.355131] ima: No TPM chip found, activating TPM-bypass! [ 3.356796] ima: Allocated hash algorithm: sha1 [ 3.358428] ima: No architecture policies found [ 3.359962] evm: Initialising EVM extended attributes: [ 3.361784] evm: security.selinux [ 3.362961] evm: security.ima [ 3.363971] evm: security.capability [ 3.365203] evm: HMAC attrs: 0x1 [ 3.367369] rtc_cmos 00:05: setting system clock to 2026-04-09 19:54:23 UTC (1775764463) [ 3.373979] debug: unmapping init [mem 0xffffffff92203000-0xffffffff923fffff] [ 3.376684] debug: unmapping init [mem 0xffffffff90f82000-0xffffffff91258fff] [ 3.385173] Write protecting the kernel read-only data: 28672k [ 3.388246] debug: unmapping init [mem 0xffffffff8f603000-0xffffffff8f7fffff] [ 3.390756] debug: unmapping init [mem 0xffffffff8ff14000-0xffffffff8fffffff] [ 3.424893] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.432350] systemd[1]: Detected virtualization kvm. [ 3.433973] systemd[1]: Detected architecture x86-64. [ 3.435694] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.461416] systemd[1]: No hostname configured. [ 3.462444] systemd[1]: Set hostname to . [ 3.464247] random: systemd: uninitialized urandom read (16 bytes read) [ 3.466612] systemd[1]: Initializing machine ID from random generator. [ 3.517052] random: ln: uninitialized urandom read (6 bytes read) [ 3.607723] random: systemd: uninitialized urandom read (16 bytes read) [ 3.610674] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.615664] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.621467] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.247162] device-mapper: uevent: version 1.0.3 [ 4.248930] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ 4.540498] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.882041] virtio_net virtio0 ens2: renamed from eth0 [ 4.940030] scsi host0: ata_piix [ 4.967501] scsi host1: ata_piix [ 4.969090] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.971681] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.195528] dracut-initqueue[594]: RTNETLINK answers: File exists [ 9.869523] random: crng init done [ 9.870477] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.204664] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.233182] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.473605] SELinux: Disabled at runtime. [ 11.534407] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.543230] systemd[1]: Detected virtualization kvm. [ 11.544869] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.048231] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.051397] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.057209] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.064498] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.068330] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.078766] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.088303] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [[ 12.192025] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.505748] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.824628] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.839094] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.953883] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.964384] EDAC sbridge: Ver: 1.1.2 [ 14.614553] Key type dns_resolver registered [ 14.920191] NFS: Registering the id_resolver key type [ 14.921970] Key type id_resolver registered [ 14.923246] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg126-server login: [ 44.410203] hrtimer: interrupt took 2379498 ns [ 55.977436] libcfs: loading out-of-tree module taints kernel. [ 56.014608] Key type ._llcrypt registered [ 56.018615] Key type .llcrypt registered [ 56.142113] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 75.638412] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 77.235407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 77.278624] alg: No test for adler32 (adler32-zlib) [ 78.894346] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 79.950803] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 81.671276] Key type lgssc registered [ 83.245249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.224265] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 147.523479] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 162.090852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 162.123860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.489749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.547745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.699863] Lustre: lustre-MDT0000: new disk, initializing [ 163.825347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.848777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.434161] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.132263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.229974] Lustre: 6514:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.253197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 184.257111] Lustre: Skipped 1 previous similar message [ 184.376925] Lustre: lustre-MDT0001: new disk, initializing [ 184.485331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.516689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 184.536197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.561339] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.201201] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.889588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.256615] Lustre: lustre-OST0000: new disk, initializing [ 205.261351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 205.330309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.477678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 210.493375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 210.551729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 212.619575] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.098800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.276380] Lustre: lustre-OST0001: new disk, initializing [ 229.280160] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.381454] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.069694] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.156414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 239.183544] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 239.325124] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 248.983582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.774792] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 269.610900] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing check_logdir /tmp/testlogs/ [ 276.374438] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing yml_node [ 280.743943] Lustre: DEBUG MARKER: Client: 2.17.51.76 [ 283.358178] Lustre: DEBUG MARKER: MDS: 2.17.51.76 [ 286.392296] Lustre: DEBUG MARKER: OSS: 2.17.51.76 [ 288.558550] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Apr 9 15:59:06 EDT 2026 [ 307.675979] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 309.912222] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 328.160758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 328.180256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.196371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.747592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.753907] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.769377] Lustre: Skipped 1 previous similar message [ 333.593767] Lustre: server umount lustre-MDT0000 complete [ 336.875417] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.905734] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 338.137961] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775764798 with bad export cookie 3450602095049032162 [ 338.141499] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.144151] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 341.984842] LustreError: 7703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.986189] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.991024] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 341.991033] Lustre: Skipped 3 previous similar messages [ 342.000257] LustreError: 7703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 342.023381] Lustre: Skipped 2 previous similar messages [ 344.032245] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.034281] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.074464] Lustre: Skipped 2 previous similar messages [ 344.969573] Lustre: server umount lustre-MDT0001 complete [ 350.150902] Lustre: server umount lustre-OST0000 complete [ 354.780413] Lustre: server umount lustre-OST0001 complete [ 361.887968] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 371.666981] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 410.128191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.481438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 410.521357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 410.667951] Lustre: lustre-MDT0000: new disk, initializing [ 410.841043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.871046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 416.431536] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 427.865989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.062416] Lustre: 15010:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 428.163249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 428.167668] Lustre: Skipped 1 previous similar message [ 428.235194] Lustre: lustre-MDT0001: new disk, initializing [ 428.348467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 428.418497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 428.440551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 433.838680] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 442.613344] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 447.099915] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 453.879737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 454.140686] Lustre: lustre-OST0000: new disk, initializing [ 454.145890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 454.265571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 455.744733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 455.774386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 455.969639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 461.859647] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 469.770930] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 474.560265] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 474.797961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 479.933672] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 480.253385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 486.880173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 486.894683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.910666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 486.930507] Lustre: Skipped 1 previous similar message [ 488.912052] Lustre: server umount lustre-OST0000 complete [ 495.077182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 495.087682] LustreError: Skipped 1 previous similar message [ 495.095594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.111095] Lustre: Skipped 1 previous similar message [ 495.117419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 508.383719] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 508.880195] Lustre: server umount lustre-MDT0000 complete [ 512.507201] LustreError: 15016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.552735] LustreError: 15016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 514.642564] LustreError: 15789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775764974 with bad export cookie 3450602095049033695 [ 514.653612] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 514.661568] LustreError: 15789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 515.377119] Lustre: server umount lustre-MDT0001 complete [ 520.377743] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:02:58 (1775764978) [ 524.206908] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 526.471813] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:03:04 (1775764984) [ 537.784244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.396815] LustreError: 19018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.553259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 545.560186] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 548.837235] LustreError: 19019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.849707] LustreError: 19019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 555.895068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.110099] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 569.149848] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 574.275209] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 583.041034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 590.947403] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 597.080247] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 626.658234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 626.682434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.721154] Lustre: Skipped 1 previous similar message [ 626.743926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 626.756302] Lustre: Skipped 6 previous similar messages [ 630.528412] Lustre: server umount lustre-OST0000 complete [ 638.433182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 638.439349] LustreError: Skipped 1 previous similar message [ 642.163686] Lustre: server umount lustre-MDT0000 complete [ 644.083136] LustreError: 19018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.108885] LustreError: 19018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 647.357976] LustreError: 18998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775765107 with bad export cookie 3450602095049034577 [ 647.384688] LustreError: 18998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 647.385723] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 647.756314] Lustre: server umount lustre-MDT0001 complete [ 662.600567] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 666.284210] Key type lgssc unregistered [ 666.500537] LNet: 22225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.509462] LNetError: 22225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 666.523154] LNet: Removed LNI 192.168.201.126@tcp [ 667.747134] Key type .llcrypt unregistered [ 667.753725] Key type ._llcrypt unregistered [ 696.525436] Key type ._llcrypt registered [ 696.532578] Key type .llcrypt registered [ 696.717218] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 698.783049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 698.806032] alg: No test for adler32 (adler32-zlib) [ 699.870680] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 700.140801] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 701.887257] Key type lgssc registered [ 703.268787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 713.690523] loop: module loaded [ 713.799061] loop0: detected capacity change from 0 to 204800000 [ 714.107215] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 714.119018] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 714.470194] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 722.807694] loop0: detected capacity change from 0 to 204800000 [ 723.194607] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.215250] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.244080] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 729.954073] loop0: detected capacity change from 0 to 204800000 [ 736.948554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 737.006585] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 738.490142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 738.550301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 738.655930] Lustre: lustre-MDT0000: new disk, initializing [ 738.774095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.807535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 743.432394] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 753.498438] loop1: detected capacity change from 0 to 204800000 [ 762.016986] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 762.466990] Lustre: lustre-OST07c6: new disk, initializing [ 762.478194] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 762.488466] Lustre: Skipped 1 previous similar message [ 762.644285] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 770.225481] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 770.260170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 770.277544] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 770.339893] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 775.712809] Lustre: 24382:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 775.726959] Lustre: 24382:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 775.742560] Lustre: 24382:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 775.748449] Lustre: 24382:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 775.760386] Lustre: 24382:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 775.765017] Lustre: 24382:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 791.674422] Lustre: 24383:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 791.684958] Lustre: 24383:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 791.691559] Lustre: 24383:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 791.700828] Lustre: 24383:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 791.715855] Lustre: 24383:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 791.721456] Lustre: 24383:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 797.763238] Lustre: 24382:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 797.789060] Lustre: 24382:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 797.800128] Lustre: 24382:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 797.812815] Lustre: 24382:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.829601] Lustre: 24382:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 797.842368] Lustre: 24382:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.850776] Lustre: 24382:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 797.857893] Lustre: 24382:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.862159] Lustre: 24382:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 797.870566] Lustre: 24382:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.880334] Lustre: 24382:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 797.890810] Lustre: 24382:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.129725] Lustre: 26048:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 801.146649] Lustre: 26048:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 801.159549] Lustre: 26048:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 801.168680] Lustre: 26048:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.177240] Lustre: 26048:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 801.192336] Lustre: 26048:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.201048] Lustre: 26048:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 801.223591] Lustre: 26048:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.243313] Lustre: 26048:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 801.258829] Lustre: 26048:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.273335] Lustre: 26048:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 801.294230] Lustre: 26048:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 810.245297] Lustre: 26048:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 810.253248] Lustre: 26048:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 3 previous similar messages [ 810.260994] Lustre: 26048:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 810.271055] Lustre: 26048:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 810.275439] Lustre: 26048:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 810.281062] Lustre: 26048:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 810.287130] Lustre: 26048:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 810.295897] Lustre: 26048:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 810.299603] Lustre: 26048:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 810.303727] Lustre: 26048:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 810.308079] Lustre: 26048:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 810.311690] Lustre: 26048:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.716395] Lustre: 24383:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 821.723219] Lustre: 24383:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 5 previous similar messages [ 821.733253] Lustre: 24383:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 821.740226] Lustre: 24383:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 821.747206] Lustre: 24383:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 821.753234] Lustre: 24383:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 821.759449] Lustre: 24383:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 821.766311] Lustre: 24383:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 821.772407] Lustre: 24383:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 821.778205] Lustre: 24383:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 821.782097] Lustre: 24383:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 821.793951] Lustre: 24383:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 872.932440] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.972484] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 876.988765] Lustre: server umount lustre-OST07c6 complete [ 888.296683] Lustre: server umount lustre-MDT0000 complete [ 912.844827] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 915.674886] Key type lgssc unregistered [ 916.019679] LNet: 27912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.036992] LNetError: 27912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.094318] LNet: Removed LNI 192.168.201.126@tcp [ 918.238405] Key type .llcrypt unregistered [ 918.240402] Key type ._llcrypt unregistered [ 940.384488] Key type ._llcrypt registered [ 940.387168] Key type .llcrypt registered [ 940.576107] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 942.093973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 942.181473] alg: No test for adler32 (adler32-zlib) [ 943.379040] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 943.850123] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 945.528951] Key type lgssc registered [ 947.170416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 957.170896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 957.193492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.073983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.793151] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 969.700701] LustreError: 29169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.731378] LustreError: 29169:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 974.832854] LustreError: 29168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.293717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.989170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 981.070888] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 987.549580] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 991.797832] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1000.000055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1000.362092] LustreError: 30763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1000.726833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.879768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1009.518717] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1016.245886] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1035.744518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1035.751193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.765497] LustreError: Skipped 1 previous similar message [ 1035.766408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1035.812981] Lustre: Skipped 1 previous similar message [ 1040.883833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1040.887372] Lustre: Skipped 2 previous similar messages [ 1041.539214] Lustre: server umount lustre-OST0000 complete [ 1047.522724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1047.525590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.555161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.104751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.121189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.109549] Lustre: server umount lustre-MDT0000 complete [ 1056.224576] LustreError: 29879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1056.243943] LustreError: 29879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1057.807939] LustreError: 29149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775765517 with bad export cookie 9711152617137873097 [ 1057.819781] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1058.231562] Lustre: server umount lustre-MDT0001 complete [ 1072.867375] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1077.108503] Key type lgssc unregistered [ 1077.549612] LNet: 32555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.558970] LNetError: 32555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.579985] LNet: Removed LNI 192.168.201.126@tcp [ 1078.730602] Key type .llcrypt unregistered [ 1078.736705] Key type ._llcrypt unregistered [ 1101.952824] Key type ._llcrypt registered [ 1101.956445] Key type .llcrypt registered [ 1102.110092] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1103.909731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1104.028364] alg: No test for adler32 (adler32-zlib) [ 1105.207695] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 1105.568561] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1107.327423] Key type lgssc registered [ 1109.044121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1119.012968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1119.038815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.647720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1126.343657] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1129.954725] LustreError: 33809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.994807] LustreError: 33809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1135.087160] LustreError: 33810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1136.443044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1136.961631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1142.882459] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1149.937959] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1154.702798] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1162.263632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1162.675114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1165.760407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1169.782600] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1176.216435] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1189.345368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1189.355172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.397902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1191.401926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1191.402080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1191.402612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.441325] Lustre: Skipped 1 previous similar message [ 1195.519828] Lustre: server umount lustre-OST0000 complete [ 1201.634049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.649261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1206.755335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1206.763799] Lustre: Skipped 2 previous similar messages [ 1206.907914] Lustre: server umount lustre-MDT0000 complete [ 1210.954398] LustreError: 33791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775765671 with bad export cookie 8014467066946099168 [ 1210.963703] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1210.968376] LustreError: 33791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1211.272739] Lustre: server umount lustre-MDT0001 complete [ 1222.959701] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1226.383457] Key type lgssc unregistered [ 1226.737679] LNet: 37009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.742865] LNetError: 37009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.818874] LNet: Removed LNI 192.168.201.126@tcp [ 1228.592933] Key type .llcrypt unregistered [ 1228.595490] Key type ._llcrypt unregistered [ 1255.435050] Key type ._llcrypt registered [ 1255.438296] Key type .llcrypt registered [ 1255.591426] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1257.242188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1257.279464] alg: No test for adler32 (adler32-zlib) [ 1258.385462] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 1258.682309] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1260.359141] Key type lgssc registered [ 1261.591780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1269.800196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1269.815813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.412542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1276.998518] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1280.993396] LustreError: 38428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.010374] LustreError: 38428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1286.115124] LustreError: 38429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.490390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.834070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1292.746259] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1299.109813] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1303.430544] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1311.092397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1311.586754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1316.717941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1318.241748] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1324.825569] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1332.194421] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1332.197658] LustreError: 39538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.202873] Lustre: Skipped 1 previous similar message [ 1332.225853] LustreError: 39538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1332.302804] Lustre: server umount lustre-MDT0000 complete [ 1336.372521] LustreError: 40045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775765796 with bad export cookie 9116616216038640651 [ 1336.376868] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1336.377093] LustreError: 40045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1337.311969] LustreError: 40096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.322993] LustreError: 40096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1337.325389] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1337.353067] Lustre: Skipped 1 previous similar message [ 1337.356774] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1338.193574] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.26@tcp (stopping) [ 1341.344377] LustreError: 38422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.347226] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1341.371387] LustreError: 38422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1342.812975] Lustre: server umount lustre-MDT0001 complete [ 1364.922699] Lustre: server umount lustre-OST0000 complete [ 1384.534245] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1387.953095] Key type lgssc unregistered [ 1388.270194] LNet: 41943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.277304] LNetError: 41943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.309131] LNet: Removed LNI 192.168.201.126@tcp [ 1389.460083] Key type .llcrypt unregistered [ 1389.465337] Key type ._llcrypt unregistered [ 1412.507346] Key type ._llcrypt registered [ 1412.511996] Key type .llcrypt registered [ 1412.625554] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1413.792107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1413.829385] alg: No test for adler32 (adler32-zlib) [ 1414.985688] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 1415.406307] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1417.095179] Key type lgssc registered [ 1418.442529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1427.222426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1427.249786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1428.942300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.083327] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1438.190162] LustreError: 43345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.221960] LustreError: 43345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1443.316183] LustreError: 43344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.893902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.375452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1448.856855] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1454.629900] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1458.155219] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1464.646974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1464.953920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.379362] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1473.283761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1479.943282] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1488.204377] Lustre: lustre-OST0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 1488.874481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1488.876787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.889736] Lustre: Skipped 1 previous similar message [ 1488.895470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1488.902035] Lustre: Skipped 1 previous similar message [ 1493.278989] Lustre: lustre-OST0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 1493.691833] Lustre: server umount lustre-OST0000 complete [ 1514.464818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.466257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1514.474869] Lustre: Skipped 1 previous similar message [ 1519.584989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.604183] Lustre: Skipped 2 previous similar messages [ 1519.908919] Lustre: server umount lustre-MDT0000 complete [ 1524.059039] LustreError: 43325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775765984 with bad export cookie 15415801970847619242 [ 1524.074547] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1530.675743] Lustre: server umount lustre-MDT0001 complete [ 1544.467571] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1547.599315] Key type lgssc unregistered [ 1547.864947] LNet: 46710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1547.872222] LNetError: 46710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1547.887740] LNet: Removed LNI 192.168.201.126@tcp [ 1548.772287] Key type .llcrypt unregistered [ 1548.775953] Key type ._llcrypt unregistered [ 1574.509859] Key type ._llcrypt registered [ 1574.511791] Key type .llcrypt registered [ 1574.589703] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1576.138094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1576.182556] alg: No test for adler32 (adler32-zlib) [ 1577.330975] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 1577.590345] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1579.335190] Key type lgssc registered [ 1580.583157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1589.477441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1589.498821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.104893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.273887] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1601.507607] LustreError: 48112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.516665] LustreError: 48112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1603.873039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.518249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1609.132846] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1615.243408] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1619.235719] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1626.141808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1626.618372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1633.169964] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1636.852441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1639.297484] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1646.598581] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1651.720788] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1664.282679] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-0916h22m01s [ 1666.300446] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1671.136734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1671.148642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.160565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1672.227703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1672.232389] Lustre: Skipped 4 previous similar messages [ 1674.077134] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 1674.083531] Lustre: Skipped 1 previous similar message [ 1674.775539] Lustre: server umount lustre-MDT0000 complete [ 1678.311960] LustreError: 48824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1678.338636] LustreError: 48824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1678.550621] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1680.173103] LustreError: 48109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1684.451923] LustreError: 50135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1684.483184] LustreError: 50135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1686.059606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.191554] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1686.307549] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.26@tcp (not set up) [ 1686.554648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.656118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1687.602975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1690.315950] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1693.778081] Lustre: DEBUG MARKER: Wait for df (36132) ... [ 1695.419930] Lustre: DEBUG MARKER: done [ 1701.343985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1701.350055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.357969] Lustre: Skipped 2 previous similar messages [ 1701.362575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1705.713738] Lustre: server umount lustre-OST0000 complete [ 1713.632655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1713.635565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1713.645226] Lustre: Skipped 2 previous similar messages [ 1713.663263] Lustre: Skipped 4 previous similar messages [ 1715.963065] Lustre: server umount lustre-MDT0000 complete [ 1719.598543] LustreError: 50404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775766179 with bad export cookie 15982050380696887132 [ 1719.608611] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1719.615150] LustreError: 50404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1719.904591] Lustre: server umount lustre-MDT0001 complete [ 1731.912971] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1734.546676] Key type lgssc unregistered [ 1734.849633] LNet: 52671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.859963] LNetError: 52671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.887362] LNet: Removed LNI 192.168.201.126@tcp [ 1735.669478] Key type .llcrypt unregistered [ 1735.671361] Key type ._llcrypt unregistered [ 1775.936837] Key type ._llcrypt registered [ 1775.938468] Key type .llcrypt registered [ 1776.001481] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1777.027825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1777.048288] alg: No test for adler32 (adler32-zlib) [ 1778.182859] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 1778.490693] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 1780.151145] Key type lgssc registered [ 1781.116611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1789.141195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1789.164916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.492921] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1790.518351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1790.705163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1795.382782] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1804.467406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1804.533232] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1804.586693] Lustre: 54864:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1804.844198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1810.004376] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1817.241904] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1821.529758] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1829.032182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1829.280886] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1829.603411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1836.367402] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 1838.710852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1843.407721] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1850.737420] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-0916h25m08s [ 1852.757737] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1861.226412] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1864.387943] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1864.471097] LustreError: 54872:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1885.066808] Lustre: lustre-MDT0000: Client bd61bd07-399c-4698-b238-2c550a9760c1 (at 192.168.201.26@tcp) reconnecting [ 1904.495208] LustreError: 54872:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1907.741785] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1910.421240] Lustre: DEBUG MARKER: done [ 1920.482640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1920.499496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.532674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1923.362199] Lustre: server umount lustre-OST0000 complete [ 1927.679932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1927.697365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.728661] Lustre: Skipped 1 previous similar message [ 1927.733469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.736278] Lustre: Skipped 1 previous similar message [ 1928.803881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1928.823422] Lustre: Skipped 2 previous similar messages [ 1932.855204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1932.865279] Lustre: Skipped 1 previous similar message [ 1933.550566] Lustre: server umount lustre-MDT0000 complete [ 1937.134604] LustreError: 56499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775766397 with bad export cookie 4407114427788345268 [ 1937.142674] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1937.143635] LustreError: 56499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1937.593411] Lustre: server umount lustre-MDT0001 complete [ 1948.456635] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 1951.033824] Key type lgssc unregistered [ 1951.289310] LNet: 58838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.297840] LNetError: 58838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.320361] LNet: Removed LNI 192.168.201.126@tcp [ 1952.082923] Key type .llcrypt unregistered [ 1952.084949] Key type ._llcrypt unregistered [ 1997.832150] Key type ._llcrypt registered [ 1997.839655] Key type .llcrypt registered [ 1997.978792] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 1999.510150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1999.560624] alg: No test for adler32 (adler32-zlib) [ 2000.689744] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 2001.030690] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2002.735606] Key type lgssc registered [ 2003.682537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2006.712587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2006.751035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.207930] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2008.228175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2008.414863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.358477] Lustre: Failing over lustre-MDT0000 [ 2009.680649] Lustre: server umount lustre-MDT0000 complete [ 2016.394912] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:27:54 (1775766474) [ 2023.078509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.573942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.287153] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2038.489736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.716746] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2038.797155] Lustre: 61625:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2039.170172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2044.042581] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2050.177138] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2054.772179] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2063.056276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2063.521710] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2063.912869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.825260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2072.033186] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2077.948971] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2084.304805] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2092.518873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.526109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2092.536201] Lustre: Skipped 2 previous similar messages [ 2092.545379] Lustre: Skipped 1 previous similar message [ 2097.633070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.641129] Lustre: Skipped 3 previous similar messages [ 2097.825604] Lustre: server umount lustre-MDT0000 complete [ 2101.716794] LustreError: 61615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775766561 with bad export cookie 8448054498103518496 [ 2101.730376] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2101.734801] LustreError: 61615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2102.156977] Lustre: server umount lustre-MDT0001 complete [ 2105.835471] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2111.334378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2118.111492] Lustre: 60549:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775766562/real 1775766562] req@0000000067f56879 x1862026092650368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775766578 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2118.132170] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.251815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.207396] Lustre: 60547:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775766566/real 1775766566] req@0000000044935f5e x1862026092650496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775766582 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2123.295176] Lustre: 60547:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775766567/real 1775766567] req@000000002ff99fe5 x1862026092650624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775766583 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2126.304565] LustreError: 60545:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a84addbd x1862026092651392/t0(0) o250->MGC192.168.201.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2126.777663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.872408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2127.327231] Lustre: 60548:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775766571/real 1775766571] req@00000000435fdc22 x1862026092650880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775766587 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2130.895773] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2131.876097] Lustre: 60549:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775766576/real 1775766576] req@00000000d2c4dcbd x1862026092651008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775766592 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2133.984979] LustreError: 65053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.016771] LustreError: 65053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2139.106710] LustreError: 63247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.124972] LustreError: 63247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2140.823356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.275618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2146.299611] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2147.032333] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2147.394875] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2154.622675] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2159.817311] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2171.874448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2171.889214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.910990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.387539] Lustre: server umount lustre-MDT0000 complete [ 2178.021331] LustreError: 65052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.044165] LustreError: 65052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2178.282659] LustreError: 61616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775766638 with bad export cookie 8448054498103520771 [ 2178.298366] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2178.565193] Lustre: server umount lustre-MDT0001 complete [ 2184.961692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2196.964660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.519171] Lustre: 60546:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775766643/real 1775766643] req@000000006fc67433 x1862026092679552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775766659 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2199.573261] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.592043] Lustre: Skipped 2 previous similar messages [ 2203.747142] LustreError: 60545:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d14434c6 x1862026092680320/t0(0) o250->MGC192.168.201.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2204.118780] LustreError: 63246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.248516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2204.304922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2208.235580] LustreError: 63247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.427080] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2219.505098] LustreError: 63246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2219.536040] LustreError: 63246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2220.316388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2225.123969] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2225.651180] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2228.776223] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2231.765949] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2236.135500] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2244.073492] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.082737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2244.095339] Lustre: Skipped 1 previous similar message [ 2244.100430] Lustre: Skipped 3 previous similar messages [ 2249.192768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2249.195508] Lustre: Skipped 4 previous similar messages [ 2250.225098] Lustre: server umount lustre-MDT0000 complete [ 2254.308408] LustreError: 63246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.331044] LustreError: 63246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2254.536148] LustreError: 61615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775766714 with bad export cookie 8448054498103522150 [ 2254.551104] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2254.551636] LustreError: 61615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2254.967638] Lustre: server umount lustre-MDT0001 complete [ 2262.245469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2266.580876] Lustre: DEBUG MARKER: files compared the same [ 2278.181418] Lustre: server umount lustre-OST0000 complete [ 2296.931339] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2301.042070] Key type lgssc unregistered [ 2301.543766] LNet: 70551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.549249] LNetError: 70551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.631832] LNet: Removed LNI 192.168.201.126@tcp [ 2303.743825] Key type .llcrypt unregistered [ 2303.745949] Key type ._llcrypt unregistered [ 2335.414604] Key type ._llcrypt registered [ 2335.418714] Key type .llcrypt registered [ 2335.554958] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2337.067025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2337.122051] alg: No test for adler32 (adler32-zlib) [ 2338.353209] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 2338.655402] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2340.375955] Key type lgssc registered [ 2341.959411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2351.895498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2351.937617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.602927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.504372] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug +malloc all [ 2363.876946] LustreError: 71953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.898249] LustreError: 71953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2367.145954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.414175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2371.921760] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug +malloc all [ 2378.210216] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2382.439139] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2390.113972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2390.411117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.333798] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug +malloc all [ 2399.670592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2403.267724] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2413.031190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2413.041230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.058850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.080814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.083426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.088324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.126600] Lustre: Skipped 1 previous similar message [ 2418.850884] Lustre: server umount lustre-OST0000 complete [ 2423.776948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2423.790263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.806430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.466619] Lustre: server umount lustre-MDT0000 complete [ 2430.431639] LustreError: 72471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.454990] LustreError: 72471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2433.669774] LustreError: 71935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775766893 with bad export cookie 360945329562471294 [ 2433.673991] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2433.681224] LustreError: 71935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2434.051955] Lustre: server umount lustre-MDT0001 complete [ 2446.458448] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2449.307848] Key type lgssc unregistered [ 2449.599609] LNet: 75134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2449.604516] LNetError: 75134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2449.628528] LNet: Removed LNI 192.168.201.126@tcp [ 2450.445539] Key type .llcrypt unregistered [ 2450.447382] Key type ._llcrypt unregistered [ 2471.656513] Key type ._llcrypt registered [ 2471.658556] Key type .llcrypt registered [ 2471.747213] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2472.988219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2473.073512] alg: No test for adler32 (adler32-zlib) [ 2474.325240] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 2474.638605] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2476.391216] Key type lgssc registered [ 2477.323722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2485.358024] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2485.370110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.961902] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2506.079414] LustreError: 76489:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.126@tcp: failed processing log, type 4: rc = -110 [ 2532.767295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.678775] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2544.876132] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2552.495492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.629090] Lustre: *** cfs_fail_loc=706, val=0*** [ 2552.971479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.036917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2557.224441] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2558.447338] LustreError: 76515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.461042] LustreError: 76515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2563.563496] LustreError: 76514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.293904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.619210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2569.696912] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2574.675860] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2578.381472] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2580.962405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2580.968551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2580.977160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2583.010230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2583.012084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.431602] Lustre: server umount lustre-OST0000 complete [ 2591.201457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2591.206244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.224062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2591.228722] Lustre: Skipped 1 previous similar message [ 2593.248787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2593.255057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.599900] Lustre: server umount lustre-MDT0000 complete [ 2598.383174] LustreError: 77465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.399883] LustreError: 77465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2600.527335] LustreError: 76496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775767060 with bad export cookie 14009401812664887642 [ 2600.541405] LustreError: 76496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2600.541489] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2600.935247] Lustre: server umount lustre-MDT0001 complete [ 2610.792340] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2613.229343] Key type lgssc unregistered [ 2613.518769] LNet: 79748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2613.523881] LNetError: 79748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2614.576379] LNet: Removed LNI 192.168.201.126@tcp [ 2615.429585] Key type .llcrypt unregistered [ 2615.433199] Key type ._llcrypt unregistered [ 2642.064974] Key type ._llcrypt registered [ 2642.067761] Key type .llcrypt registered [ 2642.301097] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2644.714784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2644.777175] alg: No test for adler32 (adler32-zlib) [ 2646.162371] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 2646.566326] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2648.276847] Key type lgssc registered [ 2649.756086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2658.675932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2658.702628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.145767] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2673.256929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2673.969195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2679.272942] LustreError: 81820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.280633] LustreError: 81820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2679.768703] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2683.361884] LustreError: 81823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.393898] LustreError: 81823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2688.490991] LustreError: 81824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2688.509735] LustreError: 81824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2690.433940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2690.885553] LustreError: 81819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.907368] LustreError: 81819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2690.918313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.172373] LustreError: 81823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.203387] LustreError: 81823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2697.541538] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2707.279280] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2707.298789] Lustre: MGS: Client b2de221c-1f9d-4d9d-aaba-e71258592bbf (at 0@lo) reconnecting [ 2707.337868] LustreError: 81834:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2707.721583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2707.810071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2712.884814] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2728.419035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2728.424220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.440297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2728.456017] Lustre: Skipped 1 previous similar message [ 2733.542602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2733.556192] Lustre: Skipped 2 previous similar messages [ 2734.135615] Lustre: server umount lustre-OST0000 complete [ 2738.662013] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2738.662873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.671043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2738.676497] Lustre: Skipped 1 previous similar message [ 2743.778590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2743.789829] Lustre: Skipped 2 previous similar messages [ 2744.381434] Lustre: server umount lustre-MDT0000 complete [ 2748.236299] Lustre: server umount lustre-MDT0001 complete [ 2751.809980] Lustre: server umount MGS complete [ 2764.759814] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:40:22 (1775767222) [ 2800.195740] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 2812.532265] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2852.275561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.259515] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2867.008849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.253211] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2867.311225] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2867.421457] Lustre: lustre-MDT0001: new disk, initializing [ 2867.544307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2867.606122] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2867.613993] Lustre: 88917:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2867.625540] LustreError: 88917:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2872.324296] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2883.221499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2883.482313] Lustre: lustre-OST0000: new disk, initializing [ 2883.486982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2883.553913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.105120] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2894.432623] Lustre: 90108:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000bb2ef822 x1862026768982528/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1775767370 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2899.766990] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2906.546940] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2906.560241] Lustre: MGS: Client a0bfb548-98fb-4b67-ab7d-0b202c97f1aa (at 0@lo) reconnecting [ 2906.642992] Lustre: 90601:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2906.670923] LustreError: 88903:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2906.680191] LustreError: 88903:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2906.696680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2906.766587] Lustre: lustre-MDT0000: new disk, initializing [ 2907.029887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2907.080135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2907.234939] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2908.204209] Lustre: 90627:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000008fe0c3ec x1862026768989056/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1775767384 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2912.524599] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 2913.332545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2913.344789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2913.392024] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2938.852230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.861478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2938.881782] Lustre: Skipped 1 previous similar message [ 2938.892235] Lustre: Skipped 1 previous similar message [ 2943.506189] Lustre: server umount lustre-OST0000 complete [ 2949.095228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.100476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2949.112465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2949.124157] Lustre: Skipped 1 previous similar message [ 2953.449510] Lustre: server umount lustre-MDT0000 complete [ 2954.210691] LustreError: 88886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2954.232632] LustreError: 88886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2963.909863] Lustre: server umount lustre-MDT0001 complete [ 2967.881795] Lustre: server umount MGS complete [ 2980.651030] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:43:59 (1775767439) [ 3003.550198] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3005.602489] Key type lgssc unregistered [ 3006.017910] LNet: 93092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3006.026698] LNetError: 93092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3006.048520] LNet: Removed LNI 192.168.201.126@tcp [ 3007.104984] Key type .llcrypt unregistered [ 3007.109360] Key type ._llcrypt unregistered [ 3032.395974] Key type ._llcrypt registered [ 3032.401120] Key type .llcrypt registered [ 3032.553547] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3033.738694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3033.887953] alg: No test for adler32 (adler32-zlib) [ 3035.089150] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 3035.580204] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3037.455186] Key type lgssc registered [ 3038.778866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3048.089243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3048.109693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.827801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.033358] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3059.175234] LustreError: 94314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.186938] LustreError: 94314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3064.291379] LustreError: 94313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.603624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.036393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3070.837615] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3077.108857] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3081.425646] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3088.022129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3088.319987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.761489] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3094.463418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3099.392093] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3109.857225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3109.863294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3109.878402] Lustre: Skipped 1 previous similar message [ 3109.909594] Lustre: Skipped 1 previous similar message [ 3114.658281] Lustre: server umount lustre-OST0000 complete [ 3120.097382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.103517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3120.112484] Lustre: Skipped 1 previous similar message [ 3120.134716] Lustre: Skipped 1 previous similar message [ 3124.921287] Lustre: server umount lustre-MDT0000 complete [ 3125.217841] LustreError: 94308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3125.230032] LustreError: 94308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3128.654215] LustreError: 94293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775767588 with bad export cookie 17169511949680122365 [ 3128.660908] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3128.667342] LustreError: 94293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3129.113903] Lustre: server umount lustre-MDT0001 complete [ 3138.948710] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3142.481438] Key type lgssc unregistered [ 3142.863575] LNet: 97493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3142.873770] LNetError: 97493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3142.891468] LNet: Removed LNI 192.168.201.126@tcp [ 3143.825429] Key type .llcrypt unregistered [ 3143.829626] Key type ._llcrypt unregistered [ 3159.128466] Key type ._llcrypt registered [ 3159.130738] Key type .llcrypt registered [ 3159.229830] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3160.717438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3160.821852] alg: No test for adler32 (adler32-zlib) [ 3161.862728] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 3162.085985] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3163.703168] Key type lgssc registered [ 3164.884174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3171.698692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3171.702212] LustreError: 98538:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 3171.962660] LustreError: 98560:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 3171.970969] LustreError: 98538:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4733 [ 3171.974809] LustreError: 98538:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3171.983298] LustreError: 98538:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3172.029639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.723428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3182.459842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.784774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3187.586411] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3194.122537] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing lsmod [ 3197.198270] LustreError: 99733:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 3197.209495] LustreError: 99734:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 3197.215764] LustreError: 99733:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 3197.218979] LustreError: 99733:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3197.228543] LustreError: 99733:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3197.234286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3197.615531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3198.674533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3203.562171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3203.572312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.594589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.074475] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.107162] Lustre: Skipped 1 previous similar message [ 3204.138252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.151890] Lustre: Skipped 2 previous similar messages [ 3207.206040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3207.230628] Lustre: Skipped 2 previous similar messages [ 3212.263634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3212.267568] Lustre: Skipped 2 previous similar messages [ 3215.327247] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3215.701412] Lustre: server umount lustre-MDT0000 complete [ 3217.377626] LustreError: 99925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3217.383985] LustreError: 99925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3219.681573] LustreError: 99036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775767679 with bad export cookie 14325135766171919284 [ 3219.690062] LustreError: 99036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3219.691796] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3221.663445] LustreError: 99029:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000319e6fd4 x1862027310030464/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3221.857156] Lustre: server umount lustre-MDT0001 complete [ 3236.002988] Lustre: server umount lustre-OST0000 complete [ 3246.055804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.490213] LustreError: 100806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3246.567884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.701735] LustreError: 100807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.756426] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3256.801408] LustreError: 100806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.707471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.951299] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3263.278069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3268.846948] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3275.649357] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3279.969510] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3287.950241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3288.342816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3292.482758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:97) [ 3294.938929] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3301.907270] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3315.180970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3315.192322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.219692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3318.247238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3318.268719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3321.117566] Lustre: server umount lustre-OST0000 complete [ 3328.484895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.501458] Lustre: Skipped 1 previous similar message [ 3331.534969] Lustre: server umount lustre-MDT0000 complete [ 3333.604163] LustreError: 100806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.623110] LustreError: 100806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3335.805362] LustreError: 100786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775767795 with bad export cookie 14325135766171920229 [ 3335.820846] LustreError: 100786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3335.824342] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3336.639145] Lustre: server umount lustre-MDT0001 complete [ 3347.611602] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3350.559172] Key type lgssc unregistered [ 3351.032430] LNet: 104002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3351.044229] LNetError: 104002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3352.100982] LNet: Removed LNI 192.168.201.126@tcp [ 3352.992247] Key type .llcrypt unregistered [ 3352.995557] Key type ._llcrypt unregistered [ 3381.850429] Key type ._llcrypt registered [ 3381.853466] Key type .llcrypt registered [ 3381.955440] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3383.617285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3384.007664] alg: No test for adler32 (adler32-zlib) [ 3385.464084] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 3385.896784] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3387.679172] Key type lgssc registered [ 3389.620123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3400.168618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3400.194896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3401.746389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3406.595689] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3410.926981] LustreError: 105406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.959972] LustreError: 105406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3416.034736] LustreError: 105407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3417.000052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3417.412276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3422.884575] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3429.464957] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3434.415377] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3441.711313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3442.001384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3446.135283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3449.100779] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3455.464824] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3463.226613] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3468.756686] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3476.961840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3476.969037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3476.974338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3476.985498] Lustre: Skipped 1 previous similar message [ 3481.153979] Lustre: server umount lustre-OST0000 complete [ 3487.202150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3487.227400] Lustre: Skipped 1 previous similar message [ 3487.242776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3487.259050] Lustre: Skipped 1 previous similar message [ 3491.336754] Lustre: server umount lustre-MDT0000 complete [ 3492.323908] LustreError: 106330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3492.351522] LustreError: 106330:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3495.539578] LustreError: 105386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775767955 with bad export cookie 17539918342512692028 [ 3495.544522] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3495.548180] LustreError: 105386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3496.104950] Lustre: server umount lustre-MDT0001 complete [ 3506.860679] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3509.773541] Key type lgssc unregistered [ 3510.062285] LNet: 108693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3510.067543] LNetError: 108693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3510.082288] LNet: Removed LNI 192.168.201.126@tcp [ 3510.837738] Key type .llcrypt unregistered [ 3510.842642] Key type ._llcrypt unregistered [ 3527.885362] Key type ._llcrypt registered [ 3527.887411] Key type .llcrypt registered [ 3528.086492] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3529.976084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3530.033118] alg: No test for adler32 (adler32-zlib) [ 3531.158605] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 3531.482545] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3533.175269] Key type lgssc registered [ 3534.299604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3542.581259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3542.611938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.498968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3549.237960] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3554.795539] LustreError: 109897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3554.846765] LustreError: 109897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3559.248538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.865088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3564.704164] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3571.389453] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3575.456142] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3582.785862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3583.099306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.275344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3592.846827] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3601.633823] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3615.201731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.245862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3615.247317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3615.284714] Lustre: Skipped 1 previous similar message [ 3617.125696] Lustre: server umount lustre-OST0000 complete [ 3625.443461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3625.446780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3625.465608] Lustre: Skipped 2 previous similar messages [ 3625.487699] Lustre: Skipped 1 previous similar message [ 3630.560972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3630.577560] Lustre: Skipped 1 previous similar message [ 3635.688040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3635.695912] Lustre: Skipped 1 previous similar message [ 3636.193106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3636.484589] Lustre: server umount lustre-MDT0000 complete [ 3640.800939] LustreError: 109894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3640.852016] LustreError: 109894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3642.433971] LustreError: 109877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775768102 with bad export cookie 2463294365931475323 [ 3642.445815] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3642.698194] Lustre: server umount lustre-MDT0001 complete [ 3657.725830] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3660.908950] Key type lgssc unregistered [ 3661.184733] LNet: 113078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3661.192184] LNetError: 113078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3661.226386] LNet: Removed LNI 192.168.201.126@tcp [ 3661.851244] Key type .llcrypt unregistered [ 3661.854278] Key type ._llcrypt unregistered [ 3686.602944] Key type ._llcrypt registered [ 3686.605838] Key type .llcrypt registered [ 3686.745478] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3687.932178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3688.114898] alg: No test for adler32 (adler32-zlib) [ 3689.195057] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 3689.491766] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3691.215975] Key type lgssc registered [ 3692.694678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3701.642539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3701.663241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.171396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3708.053304] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3713.512048] LustreError: 114481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3713.547134] LustreError: 114481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3717.964137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.233095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3723.651847] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3731.275538] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3736.852760] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3745.398308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3746.023455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3748.109865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 3754.521846] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3761.890908] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3771.205103] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3775.157727] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3775.167736] Lustre: Skipped 2 previous similar messages [ 3779.149619] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3779.203968] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3780.150024] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3780.156080] Lustre: Skipped 2 previous similar messages [ 3789.919377] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3789.933549] Lustre: Skipped 1 previous similar message [ 3797.270106] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.26@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3797.281726] Lustre: Skipped 2 previous similar messages [ 3805.828920] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.26@tcp 192.168.0.[2,10]@tcp [ 3811.857558] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.26/24@tcp in log lustre-MDT0000 [ 3811.869800] Lustre: Skipped 2 previous similar messages [ 3814.343611] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.26/24@tcp [ 3814.345933] Lustre: Skipped 1 previous similar message [ 3827.468943] Lustre: server umount lustre-OST0000 complete [ 3833.313305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.317670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3833.333017] Lustre: Skipped 1 previous similar message [ 3833.347936] Lustre: Skipped 1 previous similar message [ 3838.435462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3838.441451] Lustre: Skipped 1 previous similar message [ 3842.724568] Lustre: server umount lustre-MDT0000 complete [ 3843.557042] LustreError: 114481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3843.581077] LustreError: 114481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3844.575899] Lustre: 113791:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775768288/real 1775768288] req@000000001914ae63 x1862027862676992/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775768304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3844.621271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3844.642687] LustreError: 114478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3844.667714] LustreError: 114478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3846.496107] Lustre: 113790:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775768290/real 1775768290] req@000000002d700b63 x1862027862677376/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1775768306 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3848.049539] LustreError: 114462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775768308 with bad export cookie 10573932639292392780 [ 3848.074092] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3848.507651] Lustre: server umount lustre-MDT0001 complete [ 3860.438052] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3863.394199] Key type lgssc unregistered [ 3863.723237] LNet: 119174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3863.731923] LNetError: 119174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3863.770757] LNet: Removed LNI 192.168.201.126@tcp [ 3864.726743] Key type .llcrypt unregistered [ 3864.729767] Key type ._llcrypt unregistered [ 3896.774365] Key type ._llcrypt registered [ 3896.777525] Key type .llcrypt registered [ 3896.936486] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3898.698527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3898.741971] alg: No test for adler32 (adler32-zlib) [ 3899.986298] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 3900.275355] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3902.039475] Key type lgssc registered [ 3903.459432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3913.935883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3913.957560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.658165] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3915.694560] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.26/24@tcp [ 3915.746174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3921.373460] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3924.968972] LustreError: 120723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3924.988133] LustreError: 120723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3930.083870] LustreError: 120724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3931.561687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3931.880902] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3931.919773] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.26/24@tcp [ 3931.973359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3937.622779] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3945.030898] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3950.783957] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3961.844058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3963.012307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3971.188274] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 3978.619619] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3985.551768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:201 to 0x240000400:225) [ 3997.153453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3997.166372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3997.192764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3998.530580] Lustre: server umount lustre-OST0000 complete [ 4003.811635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4003.820599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4003.832415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4006.368423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4006.369120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.488319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4011.512082] Lustre: Skipped 2 previous similar messages [ 4013.666546] Lustre: server umount lustre-MDT0000 complete [ 4016.611661] LustreError: 120723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4016.627964] LustreError: 120723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4018.046676] LustreError: 120705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775768478 with bad export cookie 12906241714977692354 [ 4018.049644] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4018.054430] LustreError: 120705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4018.545838] Lustre: server umount lustre-MDT0001 complete [ 4029.591893] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 4032.300663] Key type lgssc unregistered [ 4032.456956] LNet: 123996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4032.467660] LNetError: 123996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4032.487430] LNet: Removed LNI 192.168.201.126@tcp [ 4033.215292] Key type .llcrypt unregistered [ 4033.217069] Key type ._llcrypt unregistered [ 4076.436507] Key type ._llcrypt registered [ 4076.438493] Key type .llcrypt registered [ 4076.530416] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 4089.512135] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 4090.578309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4090.686214] alg: No test for adler32 (adler32-zlib) [ 4091.811966] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 4092.030646] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 4093.719264] Key type lgssc registered [ 4094.443212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4128.694489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4128.712877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4129.909719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4129.948852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4130.050522] Lustre: lustre-MDT0000: new disk, initializing [ 4130.119722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4130.164517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4134.226672] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4146.688845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4146.868490] Lustre: 127363:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4146.980798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4146.985167] Lustre: Skipped 1 previous similar message [ 4147.093788] Lustre: lustre-MDT0001: new disk, initializing [ 4147.272075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4147.363237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4147.383025] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4152.490279] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4161.805958] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4167.324332] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4178.752588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4179.173152] Lustre: lustre-OST0000: new disk, initializing [ 4179.175585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4179.405166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.715224] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4187.794552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4187.809888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4188.004669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4196.206894] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4203.350538] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4203.674773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4209.385463] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4209.678516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4213.731525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4213.746643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4213.756039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4213.779868] Lustre: Skipped 1 previous similar message [ 4218.076141] Lustre: server umount lustre-OST0000 complete [ 4223.971262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4223.971781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4223.975471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4223.975480] Lustre: Skipped 1 previous similar message [ 4223.984569] Lustre: Skipped 1 previous similar message [ 4223.991395] LustreError: Skipped 1 previous similar message [ 4228.388756] Lustre: server umount lustre-MDT0000 complete [ 4232.477106] LustreError: 127355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775768692 with bad export cookie 6012894971013410044 [ 4232.490481] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4232.494583] LustreError: 127355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4232.962782] Lustre: server umount lustre-MDT0001 complete [ 4245.210984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4245.751632] LustreError: 131177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4245.772335] LustreError: 131177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4245.837160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4251.125963] LustreError: 131178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4251.553895] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4255.203677] LustreError: 131177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4260.325156] LustreError: 131178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4261.981330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4262.588179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4267.645060] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4274.462620] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4279.492799] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4288.033370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4288.522569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4294.777727] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4302.119383] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4308.727884] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4309.125896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4316.311675] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4316.629778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4326.053040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4326.335720] Lustre: lustre-OST0001: new disk, initializing [ 4326.344256] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4326.467242] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4332.178201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4332.203725] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4332.333838] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4333.400505] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4342.400615] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4342.662039] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4347.080201] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4347.308681] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4351.646985] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8aa28436d000.ost_server_uuid 50 [ 4353.440312] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8aa28436d000.ost_server_uuid in FULL state after 0 sec [ 4363.253196] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4363.260960] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.267539] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4363.276548] Lustre: Skipped 1 previous similar message [ 4363.282237] Lustre: Skipped 1 previous similar message [ 4364.993674] Lustre: server umount lustre-OST0001 complete [ 4368.369633] LustreError: 132770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.396887] LustreError: 132770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4372.448631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4372.455993] LustreError: Skipped 1 previous similar message [ 4372.457847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4372.481343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4372.505964] Lustre: Skipped 1 previous similar message [ 4375.505193] Lustre: server umount lustre-OST0000 complete [ 4380.129637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4380.133820] LustreError: Skipped 1 previous similar message [ 4380.137146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4380.143813] Lustre: Skipped 1 previous similar message [ 4380.149504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4380.153289] Lustre: Skipped 2 previous similar messages [ 4386.379815] Lustre: server umount lustre-MDT0000 complete [ 4388.855826] LustreError: 131178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4388.885827] LustreError: 131178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4391.099296] LustreError: 131159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775768851 with bad export cookie 6012894971013410919 [ 4391.105142] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4391.122811] LustreError: 131159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4392.078522] Lustre: server umount lustre-MDT0001 complete [ 4414.504734] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 4418.279377] Key type lgssc unregistered [ 4418.747664] LNet: 137020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4418.762513] LNetError: 137020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4419.815050] LNet: Removed LNI 192.168.201.126@tcp [ 4420.988583] Key type .llcrypt unregistered [ 4420.992277] Key type ._llcrypt unregistered [ 4473.614503] Key type ._llcrypt registered [ 4473.617383] Key type .llcrypt registered [ 4473.781553] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 4494.085984] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 4495.844326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4495.945049] alg: No test for adler32 (adler32-zlib) [ 4497.203329] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 4497.587353] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 4499.345848] Key type lgssc registered [ 4500.910764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4548.075177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4548.107953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4549.527862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4549.556087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4549.649393] Lustre: lustre-MDT0000: new disk, initializing [ 4549.804485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4549.826252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4555.428958] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4569.644904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.828341] Lustre: 141148:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4569.933392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4569.938966] Lustre: Skipped 1 previous similar message [ 4570.078527] Lustre: lustre-MDT0001: new disk, initializing [ 4570.186619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4570.223320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4570.237726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4576.033603] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4585.412361] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4589.595778] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4597.418106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4597.828430] Lustre: lustre-OST0000: new disk, initializing [ 4597.840511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4597.968541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4603.469020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4603.488333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4603.651067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4606.135559] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4615.148758] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4620.355319] Lustre: Failing over lustre-OST0000 [ 4620.536555] Lustre: server umount lustre-OST0000 complete [ 4637.151129] Lustre: 139368:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775769081/real 1775769081] req@00000000316512d0 x1862028709889408/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1775769097 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4637.177343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4638.240235] Lustre: 139367:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775769082/real 1775769082] req@00000000fbf27555 x1862028709890304/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775769098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4638.690975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.752472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4640.136781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4640.168626] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4641.183280] Lustre: 139369:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775769084/real 1775769084] req@00000000a989cdf7 x1862028709890560/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1775769100 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4641.211248] Lustre: 139369:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4647.270332] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4647.282717] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4648.288140] Lustre: 139367:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775769092/real 1775769092] req@000000009a12cca9 x1862028709892864/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775769108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4648.326474] Lustre: 139367:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4648.809911] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4648.810700] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4651.344204] Lustre: Failing over lustre-MDT0000 [ 4651.879501] Lustre: server umount lustre-MDT0000 complete [ 4652.017227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4652.023481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4653.087162] Lustre: 139367:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775769097/real 1775769097] req@000000002fb05b2c x1862028709893888/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775769113 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4657.632802] LustreError: 141159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4657.650122] LustreError: 141159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4658.989973] LustreError: 142898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4662.755840] LustreError: 141154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4662.772442] LustreError: 141154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4667.871781] Lustre: 139368:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775769112/real 1775769112] req@00000000cccce270 x1862028709905536/t0(0) o400->MGC192.168.201.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775769128 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4667.875428] LustreError: 142898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4667.912666] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4667.948847] LustreError: 142898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4671.972668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.993862] LustreError: 141155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4673.029049] LustreError: 141155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4678.362325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4678.433872] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4680.480518] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4683.007541] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4683.767918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4683.776331] Lustre: Skipped 1 previous similar message [ 4683.807042] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4683.890767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4693.997750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4694.014544] Lustre: Skipped 3 previous similar messages [ 4694.017879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4694.030345] Lustre: Skipped 1 previous similar message [ 4698.597904] Lustre: server umount lustre-OST0000 complete [ 4704.224170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4704.229358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4704.241514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4704.250776] Lustre: Skipped 1 previous similar message [ 4704.275690] Lustre: Skipped 1 previous similar message [ 4709.352226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4709.361053] Lustre: Skipped 1 previous similar message [ 4710.163286] Lustre: server umount lustre-MDT0000 complete [ 4714.202936] LustreError: 141138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775769174 with bad export cookie 8076063011990408489 [ 4714.207969] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4714.225475] LustreError: 141138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4714.694691] Lustre: server umount lustre-MDT0001 complete [ 4726.740726] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 4730.106854] Key type lgssc unregistered [ 4730.520489] LNet: 146392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4730.524091] LNetError: 146392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4730.547960] LNet: Removed LNI 192.168.201.126@tcp [ 4731.440969] Key type .llcrypt unregistered [ 4731.443121] Key type ._llcrypt unregistered [ 4768.374249] Key type ._llcrypt registered [ 4768.375934] Key type .llcrypt registered [ 4768.489100] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 4769.707920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4769.761885] alg: No test for adler32 (adler32-zlib) [ 4770.818877] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 4771.106627] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 4772.783235] Key type lgssc registered [ 4774.082654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4782.725737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4782.741043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.222237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4789.330952] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4794.341692] LustreError: 147898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4794.356515] LustreError: 147898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4797.320773] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4799.467066] LustreError: 147899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4801.712294] LustreError: 147927:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4801.729866] LustreError: 147927:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4801.747478] LustreError: 147927:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4802.132600] Lustre: server umount lustre-MDT0000 complete [ 4818.574620] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 4830.470238] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 4876.638851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4877.044982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4877.102625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4877.211828] Lustre: lustre-MDT0000: new disk, initializing [ 4877.325274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4877.353453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4881.902174] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4893.108686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4893.284547] Lustre: 151083:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4893.424184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4893.429956] Lustre: Skipped 1 previous similar message [ 4893.551368] Lustre: lustre-MDT0001: new disk, initializing [ 4893.819019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4893.936550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4893.963602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4898.911720] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4909.886562] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4914.967321] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4923.419504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4923.869979] Lustre: lustre-OST0000: new disk, initializing [ 4923.884614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4924.052853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4930.943905] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 4933.739768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4933.750028] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4933.863180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4938.899843] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4943.521851] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4945.684186] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 17:16:43 (1775769403) [ 5210.596519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5210.602359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5210.607351] Lustre: Skipped 1 previous similar message [ 5210.610727] Lustre: Skipped 1 previous similar message [ 5213.144309] Lustre: server umount lustre-MDT0000 complete [ 5215.716878] LustreError: 151089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5215.742768] LustreError: 151089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5217.459747] LustreError: 151860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775769677 with bad export cookie 16490664619961471829 [ 5217.463211] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5217.468933] LustreError: 151860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5220.838984] LustreError: 152831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.861503] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5220.896118] Lustre: Skipped 1 previous similar message [ 5220.898850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5222.885787] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5224.059849] Lustre: server umount lustre-MDT0001 complete [ 5243.359993] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5243.645712] Lustre: server umount lustre-OST0000 complete [ 5254.244568] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 5264.308493] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5308.360035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5308.645686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5308.681473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5308.770531] Lustre: lustre-MDT0000: new disk, initializing [ 5308.872459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5308.895813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5313.636689] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5326.414653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5326.541765] Lustre: 157344:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5326.613470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5326.618554] Lustre: Skipped 1 previous similar message [ 5326.680486] Lustre: lustre-MDT0001: new disk, initializing [ 5326.754299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5326.816544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5326.841573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5330.605502] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5341.211109] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5346.518537] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5358.990507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5359.414833] Lustre: lustre-OST0000: new disk, initializing [ 5359.426925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5359.550988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5367.733479] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5367.875965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5367.896951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5368.057587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5378.474930] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5386.044913] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5386.445513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5393.403054] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5393.778728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5399.008279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5399.026827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.033597] LustreError: Skipped 1 previous similar message [ 5399.037932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5399.102464] Lustre: Skipped 1 previous similar message [ 5403.995306] Lustre: server umount lustre-OST0000 complete [ 5409.249152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5409.250750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5409.256850] Lustre: Skipped 1 previous similar message [ 5409.270825] Lustre: Skipped 2 previous similar messages [ 5415.321973] Lustre: server umount lustre-MDT0000 complete [ 5419.444642] LustreError: 157334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775769879 with bad export cookie 16490664619961630666 [ 5419.453427] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5419.461961] LustreError: 157334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5419.834632] Lustre: server umount lustre-MDT0001 complete [ 5430.415158] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:24:48 (1775769888) [ 5450.448716] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 5461.769579] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5507.104406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5507.441428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5507.476929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5507.597044] Lustre: lustre-MDT0000: new disk, initializing [ 5507.664433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5507.677360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5513.213043] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5525.594706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.737054] Lustre: 163663:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5526.135492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5531.050238] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5538.431897] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5542.723932] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5550.490718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5550.794682] Lustre: lustre-OST0000: new disk, initializing [ 5550.798205] Lustre: Skipped 1 previous similar message [ 5550.801616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5550.814511] Lustre: Skipped 2 previous similar messages [ 5557.460760] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5557.902877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5557.910493] Lustre: Skipped 1 previous similar message [ 5557.927981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5558.032798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5565.862483] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5573.601693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5573.606403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.614092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5573.614101] Lustre: Skipped 2 previous similar messages [ 5573.650398] Lustre: Skipped 1 previous similar message [ 5579.753592] Lustre: server umount lustre-OST0000 complete [ 5587.433201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5587.440886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5590.692744] Lustre: server umount lustre-MDT0000 complete [ 5594.084987] LustreError: 163669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5594.105024] LustreError: 163669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5595.286032] LustreError: 163653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775770055 with bad export cookie 16490664619961631548 [ 5595.302688] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5596.024754] Lustre: server umount lustre-MDT0001 complete [ 5606.850937] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:27:45 (1775770065) [ 5624.070760] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 5635.634609] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5673.928114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5674.280636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5674.338192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5674.438846] Lustre: lustre-MDT0000: new disk, initializing [ 5674.599103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5674.604603] Lustre: Skipped 2 previous similar messages [ 5674.626699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5680.109930] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5692.212405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.456882] Lustre: 169590:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5692.908527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5697.517283] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5708.691900] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5715.908421] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5725.665689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5731.948213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5732.062674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5734.280474] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5744.464106] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5757.925347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5757.926800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5757.932904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5757.932915] Lustre: Skipped 6 previous similar messages [ 5757.962111] LustreError: Skipped 1 previous similar message [ 5758.013228] Lustre: Skipped 2 previous similar messages [ 5759.318059] Lustre: server umount lustre-OST0000 complete [ 5764.582429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5770.729847] Lustre: server umount lustre-MDT0000 complete [ 5773.281034] LustreError: 169599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.314554] LustreError: 169599:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5776.174272] LustreError: 169582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775770236 with bad export cookie 16490664619961632633 [ 5776.184490] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5776.202648] LustreError: 169582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5776.654485] Lustre: server umount lustre-MDT0001 complete [ 5789.698743] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5793.192977] Key type lgssc unregistered [ 5793.608937] LNet: 173072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5793.617398] LNetError: 173072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5793.661887] LNet: Removed LNI 192.168.201.126@tcp [ 5794.756236] Key type .llcrypt unregistered [ 5794.761091] Key type ._llcrypt unregistered [ 5826.609563] Key type ._llcrypt registered [ 5826.617029] Key type .llcrypt registered [ 5826.756845] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5828.601581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5828.628041] alg: No test for adler32 (adler32-zlib) [ 5829.858661] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 5830.316124] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5832.191205] Key type lgssc registered [ 5833.484857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5843.077381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5843.092192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.776678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5850.222359] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5854.187367] LustreError: 174474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5854.214522] LustreError: 174474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5859.303221] LustreError: 174475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5860.819497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5861.327294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5866.405892] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5873.217650] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5878.264487] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5885.691630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5885.926625] LustreError: 176070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5886.141386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5889.233170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5892.514569] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 5899.386142] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5920.227279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5920.229065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5920.246191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5920.280668] Lustre: Skipped 1 previous similar message [ 5925.397226] LustreError: 176072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5925.414624] LustreError: 176072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5925.556636] Lustre: server umount lustre-OST0000 complete [ 5933.044189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5933.059730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5933.093572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5933.113644] Lustre: Skipped 1 previous similar message [ 5935.585172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5935.587928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5937.147254] Lustre: server umount lustre-MDT0000 complete [ 5940.704621] LustreError: 176080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5940.736374] LustreError: 176080:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5942.784429] LustreError: 174456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775770402 with bad export cookie 962116227940947475 [ 5942.791996] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5942.797517] LustreError: 174456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5943.233524] Lustre: server umount lustre-MDT0001 complete [ 5957.890648] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 5961.737992] Key type lgssc unregistered [ 5962.099789] LNet: 177678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5962.114171] LNetError: 177678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5962.158283] LNet: Removed LNI 192.168.201.126@tcp [ 5963.161632] Key type .llcrypt unregistered [ 5963.166957] Key type ._llcrypt unregistered [ 5991.422815] Key type ._llcrypt registered [ 5991.429161] Key type .llcrypt registered [ 5991.549355] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 5993.592628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5993.691468] alg: No test for adler32 (adler32-zlib) [ 5994.833216] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 5995.101896] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 5996.816039] Key type lgssc registered [ 5998.085368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6007.493787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6007.522398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6008.989770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6013.810526] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6019.560738] LustreError: 179083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6019.600615] LustreError: 179083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6023.239276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6023.598366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6028.999468] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6035.654748] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6040.802203] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6049.606088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6050.101835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6054.205210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6057.278510] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6065.259274] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6077.752420] Lustre: lustre-OST0000: Not available for connect from 192.168.201.26@tcp (stopping) [ 6079.983147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.984975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6079.991224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6080.002904] Lustre: Skipped 1 previous similar message [ 6080.042632] Lustre: Skipped 1 previous similar message [ 6081.715824] Lustre: server umount lustre-OST0000 complete [ 6089.766877] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8aa284692000.ost_server_uuid 50 [ 6092.651081] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6110.699448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6110.699861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6110.710937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6110.728495] Lustre: Skipped 1 previous similar message [ 6110.756690] LustreError: Skipped 1 previous similar message [ 6113.458747] Lustre: server umount lustre-MDT0000 complete [ 6115.809741] LustreError: 179078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6115.833555] LustreError: 179078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6117.820303] LustreError: 179063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775770577 with bad export cookie 16227054389000899541 [ 6117.823181] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6117.834077] LustreError: 179063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6118.251533] Lustre: server umount lustre-MDT0001 complete [ 6128.778710] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:36:26 (1775770586) [ 6137.296815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6137.779148] LustreError: 182569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.883093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6142.949827] LustreError: 182570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.457667] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6147.041919] LustreError: 182569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6153.818033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.359621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6159.360061] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6167.797430] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6174.478600] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6182.538956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6182.792198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6190.135172] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6196.926675] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6204.283749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6204.620445] Lustre: lustre-OST0001: new disk, initializing [ 6204.626219] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6204.770468] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6207.886174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6213.276799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6213.301400] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6213.399165] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6213.481660] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6223.508919] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6228.448522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6228.455914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.476362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6228.484545] Lustre: Skipped 1 previous similar message [ 6234.083637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6234.098920] Lustre: Skipped 3 previous similar messages [ 6241.247415] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6241.325283] Lustre: server umount lustre-OST0000 complete [ 6244.345186] LustreError: 184167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.390254] LustreError: 184167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6249.416897] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6249.718774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6255.110760] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6255.313575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6260.513738] LustreError: 184814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6260.527240] LustreError: 184814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6275.042907] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6275.055939] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6275.059629] LustreError: Skipped 1 previous similar message [ 6275.069961] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6275.071395] Lustre: Skipped 2 previous similar messages [ 6275.120087] Lustre: Skipped 5 previous similar messages [ 6276.997778] Lustre: server umount lustre-OST0001 complete [ 6282.214782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6282.222785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6287.913627] Lustre: server umount lustre-MDT0000 complete [ 6292.562982] LustreError: 182549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775770752 with bad export cookie 16227054389000900612 [ 6292.580664] LustreError: 182549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6292.582247] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6293.019872] Lustre: server umount lustre-MDT0001 complete [ 6326.127602] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:39:43 (1775770783) [ 6335.226308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.634822] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6335.685947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6336.070524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.102343] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6353.645047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6353.739947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6353.837778] Lustre: 188372:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6359.631867] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6366.546343] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6370.707916] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6379.430214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6379.667678] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6385.356901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6387.981664] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6394.604837] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6403.267785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6403.333750] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6403.546297] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6403.554196] Lustre: Skipped 2 previous similar messages [ 6410.418164] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6417.628705] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6423.520526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6423.528541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6423.540169] Lustre: Skipped 2 previous similar messages [ 6423.548300] Lustre: Skipped 4 previous similar messages [ 6435.807236] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6435.997623] Lustre: server umount lustre-OST0000 complete [ 6438.884736] LustreError: 189994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6438.910192] LustreError: 189994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6454.245310] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6454.249697] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6454.290047] Lustre: Skipped 1 previous similar message [ 6455.077539] Lustre: server umount lustre-OST0001 complete [ 6459.367498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6459.379681] Lustre: Skipped 10 previous similar messages [ 6465.636504] Lustre: server umount lustre-MDT0000 complete [ 6470.314267] LustreError: 190014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775770930 with bad export cookie 16227054389000902012 [ 6470.316119] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6470.332586] LustreError: 190014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6471.087634] Lustre: server umount lustre-MDT0001 complete [ 6500.073765] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:42:37 (1775770957) [ 6521.357826] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 6530.641862] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 6572.758080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6573.222966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6573.280696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6573.531888] Lustre: lustre-MDT0000: new disk, initializing [ 6573.778985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6573.836090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6579.443731] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6592.355943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6592.476394] Lustre: 195985:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6592.523538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6592.528144] Lustre: Skipped 1 previous similar message [ 6592.615348] Lustre: lustre-MDT0001: new disk, initializing [ 6592.766217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6592.775119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6597.397507] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6605.499673] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6609.981354] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6617.517449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6617.956446] Lustre: lustre-OST0000: new disk, initializing [ 6617.965993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6625.389292] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6626.419674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6626.434861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6626.655811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6633.488063] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6638.442810] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6638.782129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6643.229515] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6643.445522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6647.267861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6647.284052] LustreError: Skipped 1 previous similar message [ 6647.284199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6647.292039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6647.292048] Lustre: Skipped 2 previous similar messages [ 6647.327436] Lustre: Skipped 3 previous similar messages [ 6651.678544] Lustre: server umount lustre-OST0000 complete [ 6670.303834] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6670.564079] Lustre: server umount lustre-MDT0000 complete [ 6672.869774] LustreError: 198283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6672.906873] LustreError: 198283:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 6674.684303] LustreError: 195976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775771134 with bad export cookie 16227054389000903587 [ 6674.689363] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6674.695269] LustreError: 195976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6675.089796] Lustre: server umount lustre-MDT0001 complete [ 6684.814147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6690.431678] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6699.163637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6703.707155] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6709.934358] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6714.545346] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6722.776455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6729.734796] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6735.592946] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6740.794418] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6741.086499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6746.344940] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6746.566758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6749.664928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6749.678282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6749.688403] Lustre: Skipped 2 previous similar messages [ 6754.807630] Lustre: server umount lustre-OST0000 complete [ 6761.880520] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6766.909582] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6795.772642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6800.166496] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6800.179622] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6802.426485] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6808.452621] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6815.377484] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 6815.548586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6815.596606] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6820.833921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6820.840348] LustreError: Skipped 1 previous similar message [ 6820.842752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6820.854497] Lustre: Skipped 11 previous similar messages [ 6825.856643] Lustre: server umount lustre-OST0000 complete [ 6844.383121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6844.614721] Lustre: server umount lustre-MDT0000 complete [ 6846.433642] LustreError: 199794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6846.455611] LustreError: 199794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6848.882078] LustreError: 199780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775771309 with bad export cookie 16227054389000904469 [ 6848.884343] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6848.889942] LustreError: 199780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6849.141812] Lustre: server umount lustre-MDT0001 complete [ 6857.284732] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:48:35 (1775771315) [ 6865.052899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6865.713880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6865.716975] Lustre: Skipped 6 previous similar messages [ 6870.615558] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6880.971908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6886.081163] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6892.415093] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6898.258276] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6908.185494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6917.532736] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6926.006209] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6931.383915] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6931.706402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6936.960585] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6937.202508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6944.990270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6945.312795] Lustre: lustre-OST0001: new disk, initializing [ 6945.316967] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6951.133576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6951.169100] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6951.221480] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6954.392640] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 6963.047232] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6968.589904] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6968.943315] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6974.305792] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6974.555426] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6976.993682] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6977.002836] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6977.007722] LustreError: Skipped 2 previous similar messages [ 6977.027701] Lustre: Skipped 6 previous similar messages [ 6991.327149] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6991.577319] Lustre: server umount lustre-OST0001 complete [ 6998.936302] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6999.216297] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7003.586411] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7003.796971] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7033.484627] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7038.960846] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7038.979754] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7038.983914] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7038.994233] Lustre: Skipped 1 previous similar message [ 7040.213438] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7046.400697] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7048.401554] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7056.621946] Lustre: server umount lustre-OST0001 complete [ 7079.781455] Lustre: server umount lustre-OST0000 complete [ 7085.028593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7085.034552] Lustre: Skipped 20 previous similar messages [ 7090.035414] Lustre: server umount lustre-MDT0000 complete [ 7094.004624] LustreError: 205233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775771554 with bad export cookie 16227054389000905435 [ 7094.005824] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7094.012274] LustreError: 205233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7100.854440] Lustre: server umount lustre-MDT0001 complete [ 7130.212379] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:53:08 (1775771588) [ 7139.305033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7139.945592] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7139.964519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7145.670220] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7155.690534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7155.754551] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7155.793194] Lustre: 213214:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7159.836179] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7166.068329] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7170.339458] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7177.338688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7177.502175] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7183.958642] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7189.786675] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7199.150500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7199.199906] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7207.074494] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7212.871239] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7214.958211] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7220.311108] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7220.550457] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7225.152311] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7225.386441] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7229.207200] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8aa291b49800.ost_server_uuid 50 [ 7230.736625] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8aa291b49800.ost_server_uuid in IDLE state after 0 sec [ 7231.691835] Lustre: Permanently deactivating lustre-OST0001 [ 7231.707569] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7235.496382] Lustre: Permanently reactivating lustre-OST0001 [ 7240.674402] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7240.689022] Lustre: Skipped 7 previous similar messages [ 7246.337948] Lustre: server umount lustre-OST0001 complete [ 7250.947661] LustreError: 215332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7250.969694] LustreError: 215332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 40 previous similar messages [ 7253.496461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7253.532031] LustreError: Skipped 4 previous similar messages [ 7257.874432] Lustre: server umount lustre-OST0000 complete [ 7268.534869] Lustre: server umount lustre-MDT0000 complete [ 7273.024325] LustreError: 213206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775771733 with bad export cookie 16227054389000906723 [ 7273.025239] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7273.040425] LustreError: 213206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7273.709397] Lustre: server umount lustre-MDT0001 complete [ 7302.733548] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:56:00 (1775771760) [ 7311.870677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.259560] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7312.307946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7312.316406] Lustre: Skipped 5 previous similar messages [ 7318.052606] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7326.975403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7327.059604] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7327.131370] Lustre: 219328:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7332.373155] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7339.780451] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7345.880393] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7354.782889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7355.149341] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7355.188699] Lustre: Permanently deactivating lustre-OST0000 [ 7355.223416] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7361.556562] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7362.482991] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7368.724031] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7376.869984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7376.930285] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7384.061607] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7385.354402] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7391.656340] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7393.787292] Lustre: Permanently reactivating lustre-OST0000 [ 7393.828666] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7393.840761] Lustre: Skipped 2 previous similar messages [ 7403.140390] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7403.140406] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 00000000adb8d155, cur 1775771863 deadline 1775771851 last 1775771821 [ 7403.144736] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7403.171757] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7403.227620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7415.438989] Lustre: server umount lustre-OST0001 complete [ 7434.207465] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7434.359109] Lustre: server umount lustre-OST0000 complete [ 7445.258639] Lustre: server umount lustre-MDT0000 complete [ 7449.902117] LustreError: 222471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775771910 with bad export cookie 16227054389000908368 [ 7449.902800] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7449.925364] LustreError: 222471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7450.524511] Lustre: server umount lustre-MDT0001 complete [ 7463.108582] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 7466.381542] Key type lgssc unregistered [ 7466.773513] LNet: 223742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7466.781869] LNetError: 223742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7466.820509] LNet: Removed LNI 192.168.201.126@tcp [ 7467.647446] Key type .llcrypt unregistered [ 7467.654079] Key type ._llcrypt unregistered [ 7498.546758] Key type ._llcrypt registered [ 7498.553745] Key type .llcrypt registered [ 7498.776621] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 7500.735395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7500.799359] alg: No test for adler32 (adler32-zlib) [ 7502.046817] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 7502.478367] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 7504.231218] Key type lgssc registered [ 7505.950628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7516.618419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7516.648308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7518.086978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7522.183395] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7528.426446] LustreError: 225192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7528.446257] LustreError: 225192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7533.543489] LustreError: 225193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7533.625387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7533.730766] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7533.758480] Lustre: 225180:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7533.766330] Lustre: Permanently deactivating lustre-MDT0001 [ 7533.784866] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7534.069505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7535.680297] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7535.686739] LustreError: 225223:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7535.704678] LustreError: 225223:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7535.717848] LustreError: 225223:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7539.054221] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7546.147854] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7551.106727] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7559.156088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7559.594130] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7559.599020] LustreError: 226822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7559.700092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7563.812168] LustreError: 226810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7563.859292] LustreError: 226810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7563.863944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7567.203371] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7568.876688] LustreError: 226810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7568.901590] LustreError: 226810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7573.573838] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7579.109468] LustreError: 226811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7579.149755] LustreError: 226811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7581.232529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7581.414833] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7586.817626] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7586.820351] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7588.038684] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7595.282504] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7599.259685] Lustre: Permanently reactivating lustre-MDT0001 [ 7599.290165] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7599.299830] Lustre: Skipped 2 previous similar messages [ 7602.572803] Lustre: lustre-MDT0001: Client c1d739ab-07a4-4b19-9641-b106f61fd16b (at 192.168.201.26@tcp) reconnecting [ 7608.603939] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7610.763781] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7617.874521] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7618.141587] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7624.821517] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7625.078953] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7628.174767] Lustre: Permanently deactivating lustre-MDT0001 [ 7628.191906] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7628.205196] Lustre: Skipped 2 previous similar messages [ 7635.081520] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7643.108160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7643.117899] LustreError: 225193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7643.129969] Lustre: Skipped 1 previous similar message [ 7643.186920] Lustre: server umount lustre-MDT0000 complete [ 7647.229309] LustreError: 225930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775772107 with bad export cookie 8538568025368829145 [ 7647.245044] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7647.261522] LustreError: 225930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7647.555836] Lustre: server umount lustre-MDT0001 complete [ 7652.076128] Lustre: server umount lustre-OST0000 complete [ 7665.742164] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:02:03 (1775772123) [ 7678.712982] Lustre: server umount lustre-OST0001 complete [ 7686.158114] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 7698.722868] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 7743.013618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7743.428699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7743.443298] Lustre: Skipped 2 previous similar messages [ 7743.478401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7743.624381] Lustre: lustre-MDT0000: new disk, initializing [ 7743.797774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7743.826158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7749.458381] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7762.768270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7762.992826] Lustre: 232855:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7763.094855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7763.100840] Lustre: Skipped 1 previous similar message [ 7763.241294] Lustre: lustre-MDT0001: new disk, initializing [ 7763.368810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7763.435859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7763.460813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7769.653610] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7778.683619] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7783.740931] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7792.268659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7792.780522] Lustre: lustre-OST0000: new disk, initializing [ 7792.788729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7792.932659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.372462] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7800.434711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7800.446902] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7800.660147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7809.276144] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7812.410607] LustreError: 232865:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7822.428562] LustreError: 232865:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7822.468756] LustreError: 234850:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7822.615432] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7822.900486] Lustre: lustre-OST0001: new disk, initializing [ 7822.903358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7830.206472] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7830.722708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7830.729726] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7830.845419] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7832.551091] LustreError: 234850:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7840.166021] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7842.926484] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7846.372500] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7846.384290] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7846.392154] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7846.396630] Lustre: Skipped 3 previous similar messages [ 7846.411052] Lustre: Skipped 1 previous similar message [ 7851.381284] Lustre: server umount lustre-OST0001 complete [ 7851.521604] LustreError: 235966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7851.600236] LustreError: 235966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7871.971688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7871.994503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7872.011397] Lustre: Skipped 1 previous similar message [ 7877.098490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7877.805972] Lustre: server umount lustre-OST0000 complete [ 7886.305572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7886.320305] LustreError: Skipped 1 previous similar message [ 7886.324161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7886.339564] Lustre: Skipped 1 previous similar message [ 7886.360146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7886.370249] Lustre: Skipped 1 previous similar message [ 7889.875556] Lustre: server umount lustre-MDT0000 complete [ 7894.225669] LustreError: 234608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775772354 with bad export cookie 8538568025368831749 [ 7894.233556] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7894.236098] LustreError: 234608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7900.743275] Lustre: server umount lustre-MDT0001 complete [ 7915.948587] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 7919.515514] Key type lgssc unregistered [ 7919.937606] LNet: 237505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7919.943401] LNetError: 237505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7919.968775] LNet: Removed LNI 192.168.201.126@tcp [ 7921.095223] Key type .llcrypt unregistered [ 7921.098948] Key type ._llcrypt unregistered [ 7963.661335] Key type ._llcrypt registered [ 7963.675669] Key type .llcrypt registered [ 7963.791863] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 7965.239739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7965.298139] alg: No test for adler32 (adler32-zlib) [ 7966.520087] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 7966.808458] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 7968.519167] Key type lgssc registered [ 7970.369488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7980.583738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7980.611538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7981.973974] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7982.006795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7982.289634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7988.503495] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 7997.662954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7997.766467] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7997.800403] Lustre: 239701:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7997.898044] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7998.106941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8004.244707] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8011.730271] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8017.103476] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8024.896090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8025.150580] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8025.478567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8031.353146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8032.756591] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8040.445767] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8097.760475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8097.765099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8097.780455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8098.289641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8098.300625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8102.233557] Lustre: server umount lustre-OST0000 complete [ 8106.999941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8123.081462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8123.577833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8130.549131] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8137.920278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8137.941696] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8138.029858] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8148.223331] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8148.251485] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8179.169408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8179.172293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8179.173410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8179.173417] Lustre: Skipped 2 previous similar messages [ 8179.189656] Lustre: Skipped 1 previous similar message [ 8184.291736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8184.302322] Lustre: Skipped 1 previous similar message [ 8185.413353] Lustre: server umount lustre-OST0000 complete [ 8192.492640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8192.504465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8192.527523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8192.537990] Lustre: Skipped 1 previous similar message [ 8195.856705] Lustre: server umount lustre-MDT0000 complete [ 8200.246545] LustreError: 239694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775772660 with bad export cookie 12303142766788985460 [ 8200.261647] LustreError: 239694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8200.264405] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8200.942521] Lustre: server umount lustre-MDT0001 complete [ 8214.090727] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 8217.477278] Key type lgssc unregistered [ 8217.908947] LNet: 244659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8217.911899] LNetError: 244659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8217.950859] LNet: Removed LNI 192.168.201.126@tcp [ 8219.084059] Key type .llcrypt unregistered [ 8219.086590] Key type ._llcrypt unregistered [ 8240.897449] Key type ._llcrypt registered [ 8240.899541] Key type .llcrypt registered [ 8241.012376] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 8242.024150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8242.254124] alg: No test for adler32 (adler32-zlib) [ 8243.478985] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 8243.789799] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 8245.543205] Key type lgssc registered [ 8246.895697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8255.609574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8255.657227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8257.523719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8263.298510] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8267.768732] LustreError: 246039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8267.775254] LustreError: 246039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8272.869796] LustreError: 246040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8274.123931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8274.518070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8279.249852] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8285.972655] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8290.797181] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8298.600398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8298.926563] LustreError: 247633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8299.107516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8302.226612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8306.216743] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8312.766570] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8343.521070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8343.525091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.546839] LustreError: Skipped 1 previous similar message [ 8343.549307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8343.602667] Lustre: Skipped 1 previous similar message [ 8345.777823] Lustre: server umount lustre-OST0000 complete [ 8351.203296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8351.207444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8351.213488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8351.215415] Lustre: Skipped 1 previous similar message [ 8353.763164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8353.773421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8356.458420] Lustre: server umount lustre-MDT0000 complete [ 8358.892513] LustreError: 246040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8358.933081] LustreError: 246040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8361.255044] LustreError: 247657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775772821 with bad export cookie 1179477958124559088 [ 8361.280564] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8361.290466] LustreError: 247657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8361.846905] Lustre: server umount lustre-MDT0001 complete [ 8375.028397] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 8377.698670] Key type lgssc unregistered [ 8377.940204] LNet: 249995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8377.947496] LNetError: 249995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8377.983280] LNet: Removed LNI 192.168.201.126@tcp [ 8378.906974] Key type .llcrypt unregistered [ 8378.908924] Key type ._llcrypt unregistered [ 8397.150637] Key type ._llcrypt registered [ 8397.155613] Key type .llcrypt registered [ 8397.269083] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 8398.384480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8398.421445] alg: No test for adler32 (adler32-zlib) [ 8399.703798] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 8400.034061] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 8401.799211] Key type lgssc registered [ 8403.276422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8411.511992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8411.553656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8413.200444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8419.337942] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8423.406846] LustreError: 251198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8423.432156] LustreError: 251198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8428.516558] LustreError: 251199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8429.723382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8430.430293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8436.061463] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8440.952648] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8445.759317] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8454.226575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8454.609739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8458.811972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8460.870097] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8467.042372] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8479.203894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8479.211564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8479.223137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8479.229059] Lustre: Skipped 1 previous similar message [ 8481.026605] Lustre: server umount lustre-OST0000 complete [ 8486.883349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8486.901085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8486.910498] Lustre: Skipped 1 previous similar message [ 8486.920516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8489.440849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8489.441507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8489.467101] Lustre: Skipped 1 previous similar message [ 8490.937641] Lustre: server umount lustre-MDT0000 complete [ 8494.563783] LustreError: 251910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8494.605909] LustreError: 251910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8495.108625] LustreError: 251179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775772955 with bad export cookie 153163196754875492 [ 8495.114514] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8495.123877] LustreError: 251179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8495.484910] Lustre: server umount lustre-MDT0001 complete [ 8507.035047] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 8509.355888] Key type lgssc unregistered [ 8509.630768] LNet: 254526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8509.641766] LNetError: 254526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8509.674330] LNet: Removed LNI 192.168.201.126@tcp [ 8510.829817] Key type .llcrypt unregistered [ 8510.837166] Key type ._llcrypt unregistered [ 8538.456342] Key type ._llcrypt registered [ 8538.458909] Key type .llcrypt registered [ 8538.559346] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 8539.786265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8539.944201] alg: No test for adler32 (adler32-zlib) [ 8541.265963] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 8541.588996] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 8543.407161] Key type lgssc registered [ 8545.411593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8554.106556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8554.121326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8555.573229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8560.207432] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8565.736499] LustreError: 255928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8565.755248] LustreError: 255928:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8568.237160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8568.620663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8573.418923] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8579.164627] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8583.768192] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8590.732629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8591.082552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8597.565753] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8600.310747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8604.322339] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8634.337602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8634.341958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.354351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8636.384314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8636.387627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8636.403275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8636.443467] Lustre: Skipped 1 previous similar message [ 8638.293921] Lustre: server umount lustre-OST0000 complete [ 8641.505139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8641.506741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8641.513215] Lustre: Skipped 1 previous similar message [ 8641.521225] Lustre: Skipped 1 previous similar message [ 8646.624901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8647.842104] Lustre: server umount lustre-MDT0000 complete [ 8651.747165] LustreError: 255929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8651.827655] LustreError: 255908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775773111 with bad export cookie 14235376433468404632 [ 8651.833868] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8651.836426] LustreError: 255908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8652.193199] Lustre: server umount lustre-MDT0001 complete [ 8664.884563] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 8669.133669] Key type lgssc unregistered [ 8669.554973] LNet: 259883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8669.564848] LNetError: 259883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8670.630271] LNet: Removed LNI 192.168.201.126@tcp [ 8671.847230] Key type .llcrypt unregistered [ 8671.849343] Key type ._llcrypt unregistered [ 8689.955732] Key type ._llcrypt registered [ 8689.968351] Key type .llcrypt registered [ 8690.117964] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 8691.258075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8691.382556] alg: No test for adler32 (adler32-zlib) [ 8692.418340] Lustre: Lustre: Build Version: 2.17.51_76_g43aed12 [ 8692.701084] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 8694.375237] Key type lgssc registered [ 8695.259415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8702.565611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8702.583408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8703.948293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8707.472224] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8714.211656] LustreError: 261103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.223950] LustreError: 261103:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8715.844436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8716.357226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8721.594696] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8728.210298] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8732.569912] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8738.713955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8738.984734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8744.058662] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all [ 8749.245638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8750.020439] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8764.898636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8764.902563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8764.920944] Lustre: Skipped 1 previous similar message [ 8764.948027] Lustre: Skipped 1 previous similar message [ 8769.622547] Lustre: server umount lustre-OST0000 complete [ 8777.703990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8777.710200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8777.721751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8780.772996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8780.776224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8780.786225] Lustre: Skipped 1 previous similar message [ 8781.565677] Lustre: server umount lustre-MDT0000 complete [ 8785.901530] LustreError: 261085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8785.926772] LustreError: 261085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8787.434802] LustreError: 263429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775773247 with bad export cookie 6378327985907255975 [ 8787.436111] LustreError: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8787.456893] LustreError: 263429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8788.156396] Lustre: server umount lustre-MDT0001 complete [ 8799.578827] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 8801.048574] Key type lgssc unregistered [ 8801.215373] LNet: 264451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8801.221462] LNetError: 264451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8801.232948] LNet: Removed LNI 192.168.201.126@tcp [ 8801.601162] Key type .llcrypt unregistered [ 8801.602877] Key type ._llcrypt unregistered [ 8805.993594] Key type ._llcrypt registered [ 8805.994772] Key type .llcrypt registered [ 8806.035936] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p