[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496052405 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002402] x2apic enabled [ 0.003014] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007759] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010157] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012055] SELinux: Initializing. [ 0.013088] *** VALIDATE selinux *** [ 0.023314] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029369] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032112] *** VALIDATE tmpfs *** [ 0.034119] *** VALIDATE proc *** [ 0.035267] *** VALIDATE cgroup *** [ 0.036010] *** VALIDATE cgroup2 *** [ 0.037290] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044656] debug: unmapping init [mem 0xffffffff9ba59000-0xffffffff9ba60fff] [ 0.046312] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047790] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049017] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051016] ... value mask: 0000ffffffffffff [ 0.052017] ... max period: 00007fffffffffff [ 0.053020] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.055355] rcu: Hierarchical SRCU implementation. [ 0.057541] smp: Bringing up secondary CPUs ... [ 0.058689] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.069095] smp: Brought up 1 node, 4 CPUs [ 0.071020] smpboot: Max logical packages: 1 [ 0.072012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.110785] node 0 deferred pages initialised in 37ms [ 0.114429] devtmpfs: initialized [ 0.115150] x86/mm: Memory block size: 128MB [ 0.118237] gcov: version magic: 0x41383552 [ 0.120242] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.121113] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.122303] pinctrl core: initialized pinctrl subsystem [ 0.123189] [ 0.123766] ************************************************************* [ 0.124012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.125009] ** ** [ 0.126013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.127012] ** ** [ 0.128013] ** This means that this kernel is built to expose internal ** [ 0.129013] ** IOMMU data structures, which may compromise security on ** [ 0.130064] ** your system. ** [ 0.131012] ** ** [ 0.132013] ** If you see this message and you are not debugging the ** [ 0.133011] ** kernel, report this immediately to your vendor! ** [ 0.134011] ** ** [ 0.135010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.136013] ************************************************************* [ 0.138000] NET: Registered protocol family 16 [ 0.140365] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.142041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.145058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.149099] cpuidle: using governor menu [ 0.151963] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.154362] PCI: Using configuration type 1 for base access [ 0.156164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.167057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.168014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.172211] cryptd: max_cpu_qlen set to 1000 [ 0.174727] ACPI: Added _OSI(Module Device) [ 0.176011] ACPI: Added _OSI(Processor Device) [ 0.178017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.180009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.187030] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.197290] ACPI: Interpreter enabled [ 0.199153] ACPI: PM: (supports S0 S3 S4 S5) [ 0.202022] ACPI: Using IOAPIC for interrupt routing [ 0.204273] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.207613] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.226061] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.230176] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.235024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.240078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.246900] acpiphp: Slot [2] registered [ 0.249128] acpiphp: Slot [5] registered [ 0.250118] acpiphp: Slot [6] registered [ 0.252191] acpiphp: Slot [7] registered [ 0.254235] acpiphp: Slot [8] registered [ 0.256219] acpiphp: Slot [9] registered [ 0.258139] acpiphp: Slot [10] registered [ 0.260170] acpiphp: Slot [3] registered [ 0.262172] acpiphp: Slot [4] registered [ 0.264083] acpiphp: Slot [11] registered [ 0.265098] acpiphp: Slot [12] registered [ 0.267101] acpiphp: Slot [13] registered [ 0.269106] acpiphp: Slot [14] registered [ 0.270083] acpiphp: Slot [15] registered [ 0.272102] acpiphp: Slot [16] registered [ 0.274128] acpiphp: Slot [17] registered [ 0.276267] acpiphp: Slot [18] registered [ 0.278114] acpiphp: Slot [19] registered [ 0.280103] acpiphp: Slot [20] registered [ 0.281130] acpiphp: Slot [21] registered [ 0.284107] acpiphp: Slot [22] registered [ 0.286195] acpiphp: Slot [23] registered [ 0.289176] acpiphp: Slot [24] registered [ 0.291099] acpiphp: Slot [25] registered [ 0.292090] acpiphp: Slot [26] registered [ 0.294128] acpiphp: Slot [27] registered [ 0.296118] acpiphp: Slot [28] registered [ 0.298136] acpiphp: Slot [29] registered [ 0.299149] acpiphp: Slot [30] registered [ 0.301111] acpiphp: Slot [31] registered [ 0.303062] PCI host bridge to bus 0000:00 [ 0.304019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.307023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.310022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.313023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.317022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.320019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.321160] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.327076] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.338758] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.342090] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.345016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.351019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355407] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359135] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.364049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.367000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.374015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.390023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.396016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.401010] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.428019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.451016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.467211] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.483013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.492013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.516017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.532879] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.542015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.552015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.573016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.586000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.596015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.607016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.629016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.645000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.654015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.665017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.689015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.703522] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.715016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.732015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.759018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.774512] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.778264] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.781447] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.783325] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.785273] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.790251] iommu: Default domain type: Passthrough [ 0.791000] SCSI subsystem initialized [ 0.792200] ACPI: bus type USB registered [ 0.794300] usbcore: registered new interface driver usbfs [ 0.797102] usbcore: registered new interface driver hub [ 0.799082] usbcore: registered new device driver usb [ 0.801281] pps_core: LinuxPPS API ver. 1 registered [ 0.804013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.808067] PTP clock support registered [ 0.810317] EDAC MC: Ver: 3.0.0 [ 0.811000] PCI: Using ACPI for IRQ routing [ 0.813350] NetLabel: Initializing [ 0.815022] NetLabel: domain hash size = 128 [ 0.817027] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.820104] NetLabel: unlabeled traffic allowed by default [ 0.823307] vgaarb: loaded [ 0.825624] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.828012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.836192] clocksource: Switched to clocksource kvm-clock [ 0.960028] VFS: Disk quotas dquot_6.6.0 [ 0.961311] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.963780] *** VALIDATE ramfs *** [ 0.965184] *** VALIDATE hugetlbfs *** [ 0.967426] pnp: PnP ACPI init [ 0.969673] pnp: PnP ACPI: found 6 devices [ 0.985373] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.988710] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.990531] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.992555] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.994548] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.996253] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.998815] NET: Registered protocol family 2 [ 1.002259] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.010314] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.013131] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.018385] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.021565] TCP: Hash tables configured (established 65536 bind 65536) [ 1.023806] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.026882] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.030428] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.034118] NET: Registered protocol family 1 [ 1.037090] RPC: Registered named UNIX socket transport module. [ 1.039777] RPC: Registered udp transport module. [ 1.042189] RPC: Registered tcp transport module. [ 1.044821] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.048795] NET: Registered protocol family 44 [ 1.051182] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.054990] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.058270] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.061596] PCI: CLS 0 bytes, default 64 [ 1.064398] Unpacking initramfs... [ 3.345239] debug: unmapping init [mem 0xffff959a3cc54000-0xffff959a3ffbffff] [ 3.348802] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.350621] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.353065] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.897978] Initialise system trusted keyrings [ 3.899070] Key type blacklist registered [ 3.900838] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.909148] zbud: loaded [ 3.911840] *** VALIDATE nfs *** [ 3.912658] *** VALIDATE nfs4 *** [ 3.914047] pstore: using deflate compression [ 3.916892] Platform Keyring initialized [ 4.023514] NET: Registered protocol family 38 [ 4.025613] Key type asymmetric registered [ 4.027265] Asymmetric key parser 'x509' registered [ 4.030202] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.033757] io scheduler mq-deadline registered [ 4.035654] io scheduler kyber registered [ 4.037644] io scheduler bfq registered [ 4.039569] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.042957] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.046345] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.049336] ACPI: Power Button [PWRF] [ 4.055449] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.063365] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.081290] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.091542] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.109347] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.139889] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.168969] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.174663] Non-volatile memory driver v1.3 [ 4.176672] Linux agpgart interface v0.103 [ 4.209200] virtio_blk virtio1: [vda] 134768 512-byte logical blocks (69.0 MB/65.8 MiB) [ 4.212884] vda: detected capacity change from 0 to 69001216 [ 4.250513] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.254325] vdb: detected capacity change from 0 to 1073741824 [ 4.267681] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.270789] vdc: detected capacity change from 0 to 2621440000 [ 4.309390] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.312762] vdd: detected capacity change from 0 to 2621440000 [ 4.455477] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.459125] vde: detected capacity change from 0 to 4294967296 [ 4.479185] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.482678] vdf: detected capacity change from 0 to 4294967296 [ 4.749583] libphy: Fixed MDIO Bus: probed [ 5.025279] usbcore: registered new interface driver usbserial_generic [ 5.030675] usbserial: USB Serial support registered for generic [ 5.033972] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.040845] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.043845] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.047836] mousedev: PS/2 mouse device common for all mice [ 5.051290] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.052532] rtc_cmos 00:05: RTC can wake from S4 [ 5.063587] rtc_cmos 00:05: registered as rtc0 [ 5.066354] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.072398] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.073059] intel_pstate: CPU model not supported [ 5.081321] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.086385] hid: raw HID events driver (C) Jiri Kosina [ 5.088501] usbcore: registered new interface driver usbhid [ 5.090526] usbhid: USB HID core driver [ 5.092225] drop_monitor: Initializing network drop monitor service [ 5.094849] Initializing XFRM netlink socket [ 5.096880] NET: Registered protocol family 10 [ 5.099990] Segment Routing with IPv6 [ 5.101246] NET: Registered protocol family 17 [ 5.103510] mpls_gso: MPLS GSO support [ 5.108201] RAS: Correctable Errors collector initialized. [ 5.110071] AVX version of gcm_enc/dec engaged. [ 5.111367] AES CTR mode by8 optimization enabled [ 5.214789] sched_clock: Marking stable (5214639241, 0)->(6292957993, -1078318752) [ 5.219611] registered taskstats version 1 [ 5.221810] Loading compiled-in X.509 certificates [ 5.224618] zswap: loaded using pool lzo/zbud [ 5.263381] Key type big_key registered [ 5.288937] Key type encrypted registered [ 5.293481] ima: No TPM chip found, activating TPM-bypass! [ 5.297259] ima: Allocated hash algorithm: sha1 [ 5.301327] ima: No architecture policies found [ 5.304456] evm: Initialising EVM extended attributes: [ 5.310804] evm: security.selinux [ 5.312800] evm: security.ima [ 5.315275] evm: security.capability [ 5.317356] evm: HMAC attrs: 0x1 [ 5.321408] rtc_cmos 00:05: setting system clock to 2026-05-25 05:34:37 UTC (1779687277) [ 5.333675] debug: unmapping init [mem 0xffffffff9ca03000-0xffffffff9cbfffff] [ 5.337782] debug: unmapping init [mem 0xffffffff9b782000-0xffffffff9ba58fff] [ 5.349081] Write protecting the kernel read-only data: 28672k [ 5.354639] debug: unmapping init [mem 0xffffffff99e03000-0xffffffff99ffffff] [ 5.358648] debug: unmapping init [mem 0xffffffff9a714000-0xffffffff9a7fffff] [ 5.413361] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.426274] systemd[1]: Detected virtualization kvm. [ 5.429205] systemd[1]: Detected architecture x86-64. [ 5.431638] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.471605] systemd[1]: No hostname configured. [ 5.474862] systemd[1]: Set hostname to . [ 5.477068] random: systemd: uninitialized urandom read (16 bytes read) [ 5.479897] systemd[1]: Initializing machine ID from random generator. [ 5.694412] random: systemd: uninitialized urandom read (16 bytes read) [ 5.696996] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.702198] random: systemd: uninitialized urandom read (16 bytes read) [ 5.704555] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.711529] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.567966] device-mapper: uevent: version 1.0.3 [ 6.570614] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 7.533000] random: fast init done [ 7.630414] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started udev Coldplug all Devices. [ 8.317857] scsi host0: ata_piix [ 8.375891] scsi host1: ata_piix [ 8.381084] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.383727] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 14.027257] random: crng init done [ 14.032448] random: 7 urandom warning(s) missed due to ratelimiting [ 14.145797] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.757252] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.034448] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.493686] SELinux: Disabled at runtime. [ 18.626188] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.637530] systemd[1]: Detected virtualization kvm. [ 18.639977] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.602299] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.605832] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.611158] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.617276] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.620705] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.631690] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.637041] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ 20.368989] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.290807] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.290176] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.401989] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.354267] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.473325] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit)[ 29.535225] Key type dns_resolver registered [ **] A start job is r[ 29.956574] NFS: Registering the id_resolver key type unning for Confi[ 29.958872] Key type id_resolver registered gur…only root support (10s / no limit)[ 29.961040] Key type id_legacy registered [ *] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg303-server login: [ 55.829040] hrtimer: interrupt took 1502874 ns [ 104.488888] libcfs: loading out-of-tree module taints kernel. [ 104.520891] Key type ._llcrypt registered [ 104.528454] Key type .llcrypt registered [ 104.623505] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 122.624551] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 124.027951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 124.037622] alg: No test for adler32 (adler32-zlib) [ 125.372759] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 126.038470] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 127.743212] Key type lgssc registered [ 129.202586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 144.641878] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 187.279889] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 201.889238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 201.946184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.258046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 203.304622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 203.428977] Lustre: lustre-MDT0000: new disk, initializing [ 203.607846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.632630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 208.852900] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.398140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.483669] Lustre: 6507:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 223.505368] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 223.507330] Lustre: Skipped 1 previous similar message [ 223.582660] Lustre: lustre-MDT0001: new disk, initializing [ 223.636418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.658156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 223.671250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 228.012557] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.500262] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 241.721096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.031408] Lustre: lustre-OST0000: new disk, initializing [ 242.037210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 242.049519] Lustre: 8410:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.115083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.987629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.997107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 243.094864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.979280] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.613886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.715837] Lustre: lustre-OST0001: new disk, initializing [ 261.720581] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 261.726933] Lustre: 9466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 261.778923] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 267.228117] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.384403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 271.404625] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 271.470770] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 278.359180] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 285.278572] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 291.070394] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing check_logdir /tmp/testlogs/ [ 297.325556] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing yml_node [ 302.478501] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 305.372219] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 308.848385] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 311.425966] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon May 25 01:39:41 EDT 2026 [ 334.422622] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 336.411755] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 353.248194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.257960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.271122] Lustre: Skipped 1 previous similar message [ 353.283671] Lustre: Skipped 3 previous similar messages [ 358.256801] Lustre: server umount lustre-MDT0000 complete [ 361.734582] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779687633 with bad export cookie 15374434079925470546 [ 361.739808] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 361.775602] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 363.488870] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.489812] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.490392] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 363.526133] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 363.543795] Lustre: Skipped 3 previous similar messages [ 366.496131] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.497543] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.522798] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 366.547983] Lustre: Skipped 2 previous similar messages [ 371.616084] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.621546] LustreError: 7788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.643520] LustreError: 7788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 376.287273] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 376.551375] Lustre: server umount lustre-MDT0001 complete [ 380.644881] Lustre: server umount lustre-OST0000 complete [ 384.594901] Lustre: server umount lustre-OST0001 complete [ 391.840986] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 402.278162] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 442.703880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.157185] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 443.248502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 443.355896] Lustre: lustre-MDT0000: new disk, initializing [ 443.455378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.469182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 449.232382] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 460.590297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.705712] Lustre: 14985:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 460.784247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 460.793587] Lustre: Skipped 1 previous similar message [ 460.873142] Lustre: lustre-MDT0001: new disk, initializing [ 460.990715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 461.016352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 461.026102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 464.873291] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 472.391955] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 476.717184] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 484.556229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 484.849510] Lustre: lustre-OST0000: new disk, initializing [ 484.853269] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 484.865613] Lustre: 16727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 484.962859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 486.522311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 486.542680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 486.687160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 491.630112] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 499.729402] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 503.879474] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 504.012621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 507.503414] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 507.663350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 512.490081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.492515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 512.502066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.502074] Lustre: Skipped 1 previous similar message [ 515.994899] Lustre: server umount lustre-OST0000 complete [ 522.729064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.736308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 522.742717] Lustre: Skipped 2 previous similar messages [ 522.745023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 522.750722] Lustre: Skipped 2 previous similar messages [ 522.761599] LustreError: Skipped 1 previous similar message [ 525.897057] Lustre: server umount lustre-MDT0000 complete [ 527.848653] LustreError: 15853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.867780] LustreError: 15853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 529.155603] LustreError: 14976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779687801 with bad export cookie 15374434079925472079 [ 529.158036] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.177408] LustreError: 14976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 529.501986] Lustre: server umount lustre-MDT0001 complete [ 532.963899] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:43:23 (1779687803) [ 539.573100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.959774] LustreError: 18844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.018926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 543.955127] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 550.396218] LustreError: 18845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.415685] LustreError: 18845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 552.234046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.962658] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 562.387287] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 566.060304] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 573.955975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 580.172803] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 585.272257] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 592.866398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 592.879103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.896029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 595.936829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 597.489224] Lustre: server umount lustre-OST0000 complete [ 603.616257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 603.622208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.642039] Lustre: Skipped 1 previous similar message [ 607.593903] Lustre: server umount lustre-MDT0000 complete [ 611.300589] LustreError: 18840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.316945] LustreError: 18840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 611.399473] LustreError: 18824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779687883 with bad export cookie 15374434079925472961 [ 611.402066] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.425781] LustreError: 18824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 611.680410] Lustre: server umount lustre-MDT0001 complete [ 622.859032] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 625.702581] Key type lgssc unregistered [ 625.964875] LNet: 22018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.975707] LNetError: 22018:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 625.995339] LNet: Removed LNI 192.168.203.103@tcp [ 626.877150] Key type .llcrypt unregistered [ 626.879166] Key type ._llcrypt unregistered [ 648.177714] Key type ._llcrypt registered [ 648.180378] Key type .llcrypt registered [ 648.251197] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 649.208131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 649.265116] alg: No test for adler32 (adler32-zlib) [ 650.290274] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 650.468629] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 652.183158] Key type lgssc registered [ 653.125983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 661.199401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 661.231298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.707227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.651332] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 672.230953] LustreError: 23395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.252551] LustreError: 23395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 674.207820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.547403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 678.144577] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 683.126810] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 686.746403] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 692.575391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 692.807597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 698.629546] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 704.180388] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 707.199680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 719.840775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 719.851543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.860708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 719.874761] Lustre: Skipped 1 previous similar message [ 720.609139] Lustre: server umount lustre-OST0000 complete [ 725.987517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 725.997435] LustreError: Skipped 1 previous similar message [ 726.010293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 726.034570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 726.045950] Lustre: Skipped 1 previous similar message [ 728.033316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 728.034207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.155622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 733.161222] Lustre: Skipped 2 previous similar messages [ 738.273161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 738.273937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 738.288693] Lustre: Skipped 1 previous similar message [ 738.428053] Lustre: server umount lustre-MDT0000 complete [ 741.643947] LustreError: 23375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779688013 with bad export cookie 14504257334725023463 [ 741.654624] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 741.936784] Lustre: server umount lustre-MDT0001 complete [ 751.299676] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 753.693616] Key type lgssc unregistered [ 753.930577] LNet: 27012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.938815] LNetError: 27012:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.964651] LNet: Removed LNI 192.168.203.103@tcp [ 754.685839] Key type .llcrypt unregistered [ 754.688283] Key type ._llcrypt unregistered [ 776.765382] Key type ._llcrypt registered [ 776.772226] Key type .llcrypt registered [ 776.908981] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 778.006067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 778.157688] alg: No test for adler32 (adler32-zlib) [ 779.168650] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 779.379853] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 781.023159] Key type lgssc registered [ 782.294177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 789.749636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 789.784568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.276578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 795.128497] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 801.763716] LustreError: 28410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 801.781289] LustreError: 28410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 803.238236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.659159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 807.679735] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 812.804866] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 816.527911] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 829.158983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 829.498629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 835.153781] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 836.688346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 840.433416] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 858.089453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 858.091761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.096821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 858.117681] Lustre: Skipped 1 previous similar message [ 860.080554] Lustre: server umount lustre-OST0000 complete [ 865.249253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 865.257723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.277263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 865.286931] Lustre: Skipped 1 previous similar message [ 868.322263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 868.326677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.597886] Lustre: server umount lustre-MDT0000 complete [ 873.441258] LustreError: 28410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 873.472172] LustreError: 28410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 874.566817] LustreError: 28389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779688146 with bad export cookie 12909309503755409180 [ 874.589796] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 874.919786] Lustre: server umount lustre-MDT0001 complete [ 885.989582] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 888.474273] Key type lgssc unregistered [ 888.723568] LNet: 31950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.729374] LNetError: 31950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.752057] LNet: Removed LNI 192.168.203.103@tcp [ 889.425730] Key type .llcrypt unregistered [ 889.428461] Key type ._llcrypt unregistered [ 911.212595] Key type ._llcrypt registered [ 911.217589] Key type .llcrypt registered [ 911.384796] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 912.362651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 912.474396] alg: No test for adler32 (adler32-zlib) [ 913.606749] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 913.873983] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 915.575236] Key type lgssc registered [ 916.616647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 923.738357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 923.750579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.221750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.153302] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 935.396731] LustreError: 33340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.413178] LustreError: 33340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 938.233462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.508667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 942.938788] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 948.606195] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 952.198531] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 958.746762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 959.314741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.714424] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 970.631742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 971.312345] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 980.963091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 980.967342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 980.974275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 980.994410] Lustre: Skipped 1 previous similar message [ 986.080508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 986.099176] Lustre: Skipped 2 previous similar messages [ 987.041042] Lustre: server umount lustre-OST0000 complete [ 991.201477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.204324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 991.215222] Lustre: Skipped 1 previous similar message [ 996.322442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 996.327164] Lustre: Skipped 2 previous similar messages [ 997.191488] Lustre: server umount lustre-MDT0000 complete [ 1000.312062] LustreError: 34955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779688272 with bad export cookie 5277704367431607071 [ 1000.315163] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1000.325357] LustreError: 34955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1000.635228] Lustre: server umount lustre-MDT0001 complete [ 1009.024554] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1011.399251] Key type lgssc unregistered [ 1011.693354] LNet: 36516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.709319] LNetError: 36516:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.721713] LNet: Removed LNI 192.168.203.103@tcp [ 1012.641791] Key type .llcrypt unregistered [ 1012.646822] Key type ._llcrypt unregistered [ 1034.552470] Key type ._llcrypt registered [ 1034.554740] Key type .llcrypt registered [ 1034.655851] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1035.854582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1035.906905] alg: No test for adler32 (adler32-zlib) [ 1036.888421] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 1037.059165] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1038.727168] Key type lgssc registered [ 1039.642912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1046.616570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1046.646374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.159287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.621854] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1058.791064] LustreError: 37913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1058.812787] LustreError: 37913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1059.028662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.439965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1062.825275] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1067.572894] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1071.032485] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1077.290679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1077.628374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.885904] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1088.991228] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1092.997940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1100.257858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1100.274343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.288748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1100.950372] Lustre: lustre-OST0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 1103.328639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1103.331530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.332260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1105.775365] Lustre: server umount lustre-OST0000 complete [ 1123.808768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.811059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.820597] Lustre: Skipped 1 previous similar message [ 1123.827804] Lustre: Skipped 1 previous similar message [ 1127.970156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1127.986769] Lustre: Skipped 2 previous similar messages [ 1129.985026] Lustre: server umount lustre-MDT0000 complete [ 1133.025250] LustreError: 37909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.041690] LustreError: 37909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1133.425382] LustreError: 37895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779688405 with bad export cookie 4310925615212545558 [ 1133.432954] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1133.448434] LustreError: 37895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1139.935625] Lustre: server umount lustre-MDT0001 complete [ 1151.313943] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1153.978923] Key type lgssc unregistered [ 1154.315653] LNet: 41267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.327506] LNetError: 41267:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1154.345320] LNet: Removed LNI 192.168.203.103@tcp [ 1155.148459] Key type .llcrypt unregistered [ 1155.150871] Key type ._llcrypt unregistered [ 1178.141310] Key type ._llcrypt registered [ 1178.143931] Key type .llcrypt registered [ 1178.246567] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1179.397212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1179.431083] alg: No test for adler32 (adler32-zlib) [ 1180.487274] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 1180.747155] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1182.384361] Key type lgssc registered [ 1183.353324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1190.401406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1190.418802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1191.881804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.617513] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1202.147627] LustreError: 42659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.174475] LustreError: 42659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1204.751290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.142951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1209.104609] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1214.831456] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1219.163590] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1225.918424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1226.203772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.707138] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1237.498336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1237.953265] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1244.819748] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 1246.177272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1246.184915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.215276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.716247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.718606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.910137] LustreError: 42654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.063926] Lustre: server umount lustre-MDT0000 complete [ 1252.850856] LustreError: 42660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.864922] LustreError: 42660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1254.184278] LustreError: 42642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779688526 with bad export cookie 17668206692177780800 [ 1254.189143] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1254.197382] LustreError: 42642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1255.038587] LustreError: 42654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1255.068301] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.3@tcp (stopping) [ 1255.093144] Lustre: Skipped 2 previous similar messages [ 1257.956920] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.970506] Lustre: Skipped 1 previous similar message [ 1260.000440] LustreError: 43680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.002456] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1260.036683] LustreError: 43680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1260.079022] Lustre: Skipped 1 previous similar message [ 1260.791789] Lustre: server umount lustre-MDT0001 complete [ 1291.231165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1291.346264] Lustre: server umount lustre-OST0000 complete [ 1304.590772] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1306.879517] Key type lgssc unregistered [ 1307.129525] LNet: 46164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1307.136258] LNetError: 46164:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1307.145638] LNet: Removed LNI 192.168.203.103@tcp [ 1307.885741] Key type .llcrypt unregistered [ 1307.890841] Key type ._llcrypt unregistered [ 1329.995480] Key type ._llcrypt registered [ 1329.999922] Key type .llcrypt registered [ 1330.108494] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1331.252613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1331.427787] alg: No test for adler32 (adler32-zlib) [ 1332.490816] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 1332.750254] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1334.447513] Key type lgssc registered [ 1336.108747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1344.148388] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1344.168818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1359.776527] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1364.895286] LustreError: 47530:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.103@tcp: failed processing log, type 4: rc = -110 [ 1391.521065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.778216] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1401.123853] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1567.528193] Lustre: server umount lustre-OST0000 complete [ 1583.219752] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1585.529548] Key type lgssc unregistered [ 1585.797489] LNet: 49075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.805347] LNetError: 49075:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1585.821301] LNet: Removed LNI 192.168.203.103@tcp [ 1586.514288] Key type .llcrypt unregistered [ 1586.516282] Key type ._llcrypt unregistered [ 1606.282065] Key type ._llcrypt registered [ 1606.286041] Key type .llcrypt registered [ 1606.442130] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1607.482292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1607.507782] alg: No test for adler32 (adler32-zlib) [ 1608.486751] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 1608.712096] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1610.415489] Key type lgssc registered [ 1611.616431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1620.558847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1620.574561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1622.166133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.498479] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1632.747286] LustreError: 50466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1632.771891] LustreError: 50466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1635.958068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1636.469864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1640.575141] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1645.673910] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1650.265311] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1657.683810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1658.223395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.667401] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1667.442857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1669.987943] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1672.677966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1672.693624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1672.701730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1672.710783] Lustre: Skipped 1 previous similar message [ 1673.996805] Lustre: server umount lustre-OST0000 complete [ 1677.281359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1677.295520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1677.312750] Lustre: Skipped 1 previous similar message [ 1677.326243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1682.916423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1682.932692] Lustre: Skipped 2 previous similar messages [ 1683.453907] Lustre: server umount lustre-MDT0000 complete [ 1687.062052] LustreError: 50445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779688959 with bad export cookie 11143497150645683172 [ 1687.070236] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1687.076049] LustreError: 50445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1687.593901] Lustre: server umount lustre-MDT0001 complete [ 1699.617885] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1702.300446] Key type lgssc unregistered [ 1702.547683] LNet: 53644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.552733] LNetError: 53644:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.566969] LNet: Removed LNI 192.168.203.103@tcp [ 1703.421146] Key type .llcrypt unregistered [ 1703.426428] Key type ._llcrypt unregistered [ 1723.062370] Key type ._llcrypt registered [ 1723.064716] Key type .llcrypt registered [ 1723.174817] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1723.887446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1723.903533] alg: No test for adler32 (adler32-zlib) [ 1724.942298] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 1725.121310] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1726.791241] Key type lgssc registered [ 1727.717910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1733.718071] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1733.731385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1749.279777] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1754.399372] LustreError: 54995:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.103@tcp: failed processing log, type 4: rc = -110 [ 1781.087176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1787.101485] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1793.186606] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1799.979488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.594672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.662284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1804.285554] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1805.793155] LustreError: 55021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1805.820950] LustreError: 55021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1810.918785] LustreError: 55020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1812.656798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1812.941607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1816.911160] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1821.796454] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1824.842457] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1828.321256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1828.326266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.338567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1830.377192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.383275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1830.403888] Lustre: Skipped 1 previous similar message [ 1832.674324] Lustre: server umount lustre-OST0000 complete [ 1853.921253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1853.929729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.957695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1855.971120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1855.984414] Lustre: Skipped 3 previous similar messages [ 1858.913405] Lustre: server umount lustre-MDT0000 complete [ 1861.092759] LustreError: 57388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1861.116797] LustreError: 57388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1862.834361] LustreError: 55001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779689135 with bad export cookie 5537048774795531174 [ 1862.846515] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1862.849830] LustreError: 55001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1869.365570] Lustre: server umount lustre-MDT0001 complete [ 1879.701812] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1882.426719] Key type lgssc unregistered [ 1882.799544] LNet: 58383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.806476] LNetError: 58383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.823941] LNet: Removed LNI 192.168.203.103@tcp [ 1883.435589] Key type .llcrypt unregistered [ 1883.445423] Key type ._llcrypt unregistered [ 1904.166417] Key type ._llcrypt registered [ 1904.168084] Key type .llcrypt registered [ 1904.282544] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1905.362992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1905.379510] alg: No test for adler32 (adler32-zlib) [ 1906.338487] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 1906.477793] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1908.096174] Key type lgssc registered [ 1909.127830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1916.394819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1916.418159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.974830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.003695] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1928.166190] LustreError: 59761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1928.193499] LustreError: 59761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1931.015753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.460405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1935.636194] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1941.533540] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1945.580614] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1951.861901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1952.188418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.197165] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1962.629422] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1963.440093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1989.092803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1989.108665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1989.135668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1994.209465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1994.227854] Lustre: Skipped 2 previous similar messages [ 1994.871926] Lustre: server umount lustre-OST0000 complete [ 2003.424660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2003.449725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.482784] Lustre: Skipped 1 previous similar message [ 2003.494073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2006.015822] Lustre: server umount lustre-MDT0000 complete [ 2009.568606] LustreError: 61922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2011.369208] LustreError: 59742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779689283 with bad export cookie 1698459559901713648 [ 2011.377549] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2011.385124] LustreError: 59742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2011.849309] Lustre: server umount lustre-MDT0001 complete [ 2023.208815] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2026.397127] Key type lgssc unregistered [ 2026.768565] LNet: 62961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2026.777840] LNetError: 62961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2026.792629] LNet: Removed LNI 192.168.203.103@tcp [ 2027.818179] Key type .llcrypt unregistered [ 2027.828052] Key type ._llcrypt unregistered [ 2052.313173] Key type ._llcrypt registered [ 2052.315584] Key type .llcrypt registered [ 2052.401332] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2053.499118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2053.533922] alg: No test for adler32 (adler32-zlib) [ 2054.537757] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 2054.791570] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2056.503681] Key type lgssc registered [ 2057.659477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2065.742419] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 02:08:56 (1779689336) [ 2071.525038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2071.553517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.007832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.819685] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2083.307646] LustreError: 64647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.325586] LustreError: 64647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2087.148097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2087.715788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2092.387437] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2098.136503] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2102.467103] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2109.494428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2109.793821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.191880] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2118.001889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2119.763852] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2126.037724] Lustre: Failing over lustre-MDT0000 [ 2126.321751] Lustre: server umount lustre-MDT0000 complete [ 2127.516117] LustreError: 64641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.356165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2128.364560] Lustre: Skipped 2 previous similar messages [ 2132.596995] LustreError: 64642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.626398] LustreError: 64642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2134.575375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.666851] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2134.803487] Lustre: *** cfs_fail_loc=135, val=0*** [ 2134.876847] LustreError: 67393:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2134.883568] LustreError: 67393:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.103@tcp: cfg command failed: rc = -2 [ 2134.888670] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2134.888670] [ 2134.898117] LustreError: MGC192.168.203.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2134.906947] LustreError: 67381:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2134.911335] LustreError: 67381:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2134.918851] LustreError: 67381:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2134.923630] LustreError: 67381:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2134.948280] Lustre: server umount lustre-MDT0000 complete [ 2134.951821] LustreError: 67381:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2137.719020] LustreError: 66378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.737837] LustreError: 66378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2141.500940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.659087] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2141.678955] LustreError: 63713:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000498affe6 x1866139514728704/t0(0) o250->MGC192.168.203.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2141.699321] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x182845b3d75488d [ 2141.707952] Lustre: MGC192.168.203.103@tcp: Connection restored to 0@lo (at 0@lo) [ 2141.907459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.966796] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2142.845991] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2146.516904] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2147.308667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2147.335585] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2147.397323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2157.544063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.555761] Lustre: Skipped 1 previous similar message [ 2157.558113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2157.577065] Lustre: Skipped 1 previous similar message [ 2160.950283] Lustre: server umount lustre-OST0000 complete [ 2167.776709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.804906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2167.822362] Lustre: Skipped 1 previous similar message [ 2171.553682] Lustre: server umount lustre-MDT0000 complete [ 2172.903228] LustreError: 66165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.924172] LustreError: 66165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2175.467820] LustreError: 64629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779689447 with bad export cookie 108794868418300045 [ 2175.491768] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2175.740772] Lustre: server umount lustre-MDT0001 complete [ 2185.488898] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2188.219987] Key type lgssc unregistered [ 2188.515908] LNet: 69138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2188.525404] LNetError: 69138:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.543323] LNet: Removed LNI 192.168.203.103@tcp [ 2190.518843] Key type .llcrypt unregistered [ 2190.524818] Key type ._llcrypt unregistered [ 2212.817290] Key type ._llcrypt registered [ 2212.820236] Key type .llcrypt registered [ 2213.013112] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2214.063765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2214.124578] alg: No test for adler32 (adler32-zlib) [ 2215.399224] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 2215.715736] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2217.439231] Key type lgssc registered [ 2218.460845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2227.612899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2227.655045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.398868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.661791] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2238.947356] LustreError: 70528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.972785] LustreError: 70528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2243.414520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.804928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2248.172393] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2252.957436] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2256.020305] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2262.961239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2263.452591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.929577] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2274.808823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2274.865180] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2281.440218] Lustre: Failing over lustre-MDT0000 [ 2281.676055] Lustre: server umount lustre-MDT0000 complete [ 2282.640372] LustreError: 70524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.025246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.036289] LustreError: 71242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.038384] Lustre: Skipped 1 previous similar message [ 2285.075606] LustreError: 71242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2287.744637] LustreError: 70524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2290.840843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.929396] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2291.046384] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2291.172189] LustreError: 73277:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2291.181354] LustreError: 73277:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.103@tcp: cfg command failed: rc = -9 [ 2291.196240] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2291.196240] [ 2291.203996] LustreError: MGC192.168.203.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2291.216380] LustreError: 73266:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2291.224337] LustreError: 73266:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 2291.230615] LustreError: 73266:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2291.234705] LustreError: 73266:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2291.265672] Lustre: server umount lustre-MDT0000 complete [ 2291.268781] LustreError: 73266:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2292.856159] LustreError: 71551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.868970] LustreError: 71551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2297.772972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.887583] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2297.901566] LustreError: 70520:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2297.978897] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.3@tcp (not set up) [ 2298.199086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.271760] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2302.406467] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2303.091910] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2303.475500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2303.519830] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2303.575673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2309.089427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2309.096376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.110536] Lustre: Skipped 1 previous similar message [ 2309.117420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2309.601528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2311.187978] Lustre: server umount lustre-OST0000 complete [ 2318.821759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2318.832363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2318.837708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2318.843569] Lustre: Skipped 2 previous similar messages [ 2318.855872] Lustre: Skipped 2 previous similar messages [ 2321.234215] Lustre: server umount lustre-MDT0000 complete [ 2323.936101] LustreError: 70528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.968692] LustreError: 70528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2324.910710] LustreError: 74291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779689597 with bad export cookie 6450639643463353686 [ 2324.912689] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2324.921080] LustreError: 74291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2325.317212] Lustre: server umount lustre-MDT0001 complete [ 2334.924961] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2337.520155] Key type lgssc unregistered [ 2337.823260] LNet: 75025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.842294] LNetError: 75025:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2337.865710] LNet: Removed LNI 192.168.203.103@tcp [ 2338.542209] Key type .llcrypt unregistered [ 2338.544401] Key type ._llcrypt unregistered [ 2349.825552] Key type ._llcrypt registered [ 2349.827400] Key type .llcrypt registered [ 2349.911119] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2350.105838] Key type .llcrypt unregistered [ 2350.108494] Key type ._llcrypt unregistered [ 2370.585749] Key type ._llcrypt registered [ 2370.588706] Key type .llcrypt registered [ 2370.694288] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2371.403526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2371.467465] alg: No test for adler32 (adler32-zlib) [ 2372.441681] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 2372.629049] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2374.303169] Key type lgssc registered [ 2375.332461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2381.285893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2381.307193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.680148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.562430] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2392.040474] LustreError: 76874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.053646] LustreError: 76874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2393.991107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.255653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2397.755628] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2402.465060] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2406.118784] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2412.391061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2412.698433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.569383] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2422.426395] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2427.057057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2437.603386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2437.604498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2437.606799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2437.624459] Lustre: Skipped 1 previous similar message [ 2442.224563] Lustre: server umount lustre-OST0000 complete [ 2447.841555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2447.844800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2447.855805] Lustre: Skipped 1 previous similar message [ 2447.863466] Lustre: Skipped 1 previous similar message [ 2451.644771] Lustre: server umount lustre-MDT0000 complete [ 2452.960831] LustreError: 78932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.981100] LustreError: 78932:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2455.270461] LustreError: 76853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779689727 with bad export cookie 17900176606459763906 [ 2455.276079] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2455.284349] LustreError: 76853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2455.692921] Lustre: server umount lustre-MDT0001 complete [ 2464.954886] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2467.459670] Key type lgssc unregistered [ 2467.776627] LNet: 80068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2467.787734] LNetError: 80068:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.803612] LNet: Removed LNI 192.168.203.103@tcp [ 2468.580247] Key type .llcrypt unregistered [ 2468.581897] Key type ._llcrypt unregistered [ 2486.184471] Key type ._llcrypt registered [ 2486.186698] Key type .llcrypt registered [ 2486.272962] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2487.041640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2487.161266] alg: No test for adler32 (adler32-zlib) [ 2488.140978] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 2488.351770] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2489.999157] Key type lgssc registered [ 2490.976854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2496.779994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2496.810314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.202216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.094599] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2507.619752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.872640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2510.820507] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2515.511572] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2518.853432] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2524.577649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2524.930090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2529.853441] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2533.593936] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2544.374442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2549.735768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.736650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2549.740708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.745349] Lustre: Skipped 1 previous similar message [ 2551.377308] Lustre: server umount lustre-OST0000 complete [ 2554.848529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.865122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2554.873972] Lustre: Skipped 2 previous similar messages [ 2559.971578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2560.397967] Lustre: server umount lustre-MDT0000 complete [ 2562.987661] LustreError: 81422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779689835 with bad export cookie 18261480799254230138 [ 2562.990845] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2562.997123] LustreError: 81422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2563.312169] Lustre: server umount lustre-MDT0001 complete [ 2571.530871] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2573.661853] Key type lgssc unregistered [ 2573.905135] LNet: 84638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2573.913841] LNetError: 84638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2573.934763] LNet: Removed LNI 192.168.203.103@tcp [ 2574.522390] Key type .llcrypt unregistered [ 2574.524597] Key type ._llcrypt unregistered [ 2590.258633] Key type ._llcrypt registered [ 2590.262491] Key type .llcrypt registered [ 2590.340174] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2590.877054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2590.890799] alg: No test for adler32 (adler32-zlib) [ 2591.930556] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 2592.143300] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2593.815323] Key type lgssc registered [ 2594.779625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2600.188430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2600.212601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.558337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2604.578953] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2611.680078] LustreError: 86013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.702486] LustreError: 86013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2611.815392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2612.067685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2615.640802] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2620.356636] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2623.512762] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2628.891083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2629.119839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2633.109871] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2637.263228] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2646.511423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2656.739169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2656.741128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2656.746136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2656.771814] Lustre: Skipped 1 previous similar message [ 2658.248766] Lustre: server umount lustre-OST0000 complete [ 2661.858189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.866025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2661.877176] Lustre: Skipped 1 previous similar message [ 2661.882365] Lustre: Skipped 1 previous similar message [ 2666.977965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2666.982946] Lustre: Skipped 1 previous similar message [ 2667.767496] Lustre: server umount lustre-MDT0000 complete [ 2671.200367] LustreError: 88176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779689943 with bad export cookie 6084776086771586821 [ 2671.215419] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2671.223496] LustreError: 88176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2671.540266] Lustre: server umount lustre-MDT0001 complete [ 2680.644422] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2682.686503] Key type lgssc unregistered [ 2682.895484] LNet: 89212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2682.908152] LNetError: 89212:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2682.930291] LNet: Removed LNI 192.168.203.103@tcp [ 2683.570170] Key type .llcrypt unregistered [ 2683.573391] Key type ._llcrypt unregistered [ 2702.528523] Key type ._llcrypt registered [ 2702.530761] Key type .llcrypt registered [ 2702.647607] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2703.590987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2703.630211] alg: No test for adler32 (adler32-zlib) [ 2704.880833] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 2705.142624] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2706.855847] Key type lgssc registered [ 2707.958840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2714.922540] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2714.948919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2730.591401] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2735.711235] LustreError: 90561:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.103@tcp: failed processing log, type 4: rc = -110 [ 2762.399183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2766.147820] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2769.601269] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2773.568762] Lustre: server umount lustre-OST0000 complete [ 2779.368342] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:20:50 (1779690050) [ 2783.761944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.085272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2787.315037] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2789.346919] LustreError: 92084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2789.368084] LustreError: 92084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2794.331401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.466561] LustreError: 92085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2794.715699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2797.910694] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2802.766535] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2806.244312] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2812.498674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2812.777061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.492458] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2822.402179] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2829.175505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2844.643524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2844.647357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.677454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.689160] Lustre: Skipped 1 previous similar message [ 2849.276613] Lustre: server umount lustre-OST0000 complete [ 2854.893958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.902593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.911912] Lustre: Skipped 2 previous similar messages [ 2860.000694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2860.006662] Lustre: Skipped 2 previous similar messages [ 2864.098352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2864.105994] Lustre: Skipped 1 previous similar message [ 2867.167136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2867.320375] Lustre: server umount lustre-MDT0000 complete [ 2869.217336] LustreError: 92085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.233568] LustreError: 92085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2870.442433] LustreError: 92064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690142 with bad export cookie 12012167281802432898 [ 2870.452614] LustreError: 92064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2870.453382] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2870.704757] Lustre: server umount lustre-MDT0001 complete [ 2879.786390] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2881.966632] Key type lgssc unregistered [ 2882.260382] LNet: 95441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2882.265341] LNetError: 95441:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2882.288153] LNet: Removed LNI 192.168.203.103@tcp [ 2883.028986] Key type .llcrypt unregistered [ 2883.031315] Key type ._llcrypt unregistered [ 2919.037371] Key type ._llcrypt registered [ 2919.042933] Key type .llcrypt registered [ 2919.156976] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2920.200728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2920.220112] alg: No test for adler32 (adler32-zlib) [ 2921.296280] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 2921.526363] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2923.247168] Key type lgssc registered [ 2924.244642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2930.505618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2930.522295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.809327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.801873] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2940.896651] LustreError: 97607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.912082] LustreError: 97607:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2941.976926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2942.394614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2946.201811] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2951.432735] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2954.563655] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2960.594532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2960.899775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.748958] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2970.672822] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2976.302790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2982.368778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2982.390453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.406862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2986.464095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2986.471805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2986.475151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2986.500788] Lustre: Skipped 1 previous similar message [ 2988.328777] Lustre: server umount lustre-OST0000 complete [ 2991.586368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.590064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.600249] Lustre: Skipped 1 previous similar message [ 2996.704292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2996.707845] Lustre: Skipped 1 previous similar message [ 2997.489347] Lustre: server umount lustre-MDT0000 complete [ 3000.518846] LustreError: 97587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690272 with bad export cookie 2209877064056189796 [ 3000.521869] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3000.533703] LustreError: 97587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3000.911978] Lustre: server umount lustre-MDT0001 complete [ 3009.907225] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 3012.215417] Key type lgssc unregistered [ 3012.477052] LNet: 100784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3012.484780] LNetError: 100784:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3012.503701] LNet: Removed LNI 192.168.203.103@tcp [ 3013.060246] Key type .llcrypt unregistered [ 3013.062544] Key type ._llcrypt unregistered [ 3027.855187] Key type ._llcrypt registered [ 3027.858193] Key type .llcrypt registered [ 3028.001936] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3028.752601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3028.852074] alg: No test for adler32 (adler32-zlib) [ 3029.856320] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 3030.045250] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 3031.687221] Key type lgssc registered [ 3032.554325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3038.914632] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3038.935990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3054.495327] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3059.615230] LustreError: 101991:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.103@tcp: failed processing log, type 4: rc = -110 [ 3086.303247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3091.566206] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3096.553894] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3102.025716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3102.388810] LustreError: MGC192.168.203.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3102.395731] LustreError: 102902:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3102.400226] LustreError: 102902:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3102.409978] LustreError: 102902:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3102.465860] Lustre: server umount lustre-MDT0000 complete [ 3102.467783] LustreError: 102902:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3123.168786] Lustre: 102003:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779690379/real 1779690379] req@00000000740c6cb8 x1866140538113408/t0(0) o101->MGC192.168.203.103@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1779690395 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3123.197782] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3123.216557] LustreError: 102003:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3143.718932] Lustre: server umount lustre-OST0000 complete [ 3150.725368] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3156.610878] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3181.226383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.407949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3181.430956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3181.510387] Lustre: lustre-MDT0000: new disk, initializing [ 3181.569469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3181.580292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3185.040743] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3192.902422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3192.992728] Lustre: 105659:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3193.032301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3193.035343] Lustre: Skipped 1 previous similar message [ 3193.097214] Lustre: lustre-MDT0001: new disk, initializing [ 3193.185022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3193.221454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3193.234351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3196.468035] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3202.539862] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3205.443037] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3210.560337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3210.733572] Lustre: lustre-OST0000: new disk, initializing [ 3210.737474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3210.742981] Lustre: 107399:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3210.824737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3215.338468] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3218.473534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3218.483442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3218.547993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3221.038079] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3224.783175] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3225.002632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3228.674953] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3228.874249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3233.760265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3233.771150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3233.790908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3234.276395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3234.288693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3234.313624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3234.317689] Lustre: Skipped 1 previous similar message [ 3236.493389] Lustre: server umount lustre-OST0000 complete [ 3239.392451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.403235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3239.406716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3244.513233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3244.523031] Lustre: Skipped 2 previous similar messages [ 3245.536555] Lustre: server umount lustre-MDT0000 complete [ 3248.264376] LustreError: 105653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690520 with bad export cookie 17154421988513516675 [ 3248.269601] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3248.275280] LustreError: 105653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3248.529105] Lustre: server umount lustre-MDT0001 complete [ 3254.771611] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:28:45 (1779690525) [ 3257.062958] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3270.097610] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3277.002402] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3301.770112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.941865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3301.962680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3302.022752] Lustre: lustre-MDT0000: new disk, initializing [ 3302.088756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3302.109636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3305.265613] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3312.361300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.432182] Lustre: 112115:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3312.466959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3312.474028] Lustre: Skipped 1 previous similar message [ 3312.533043] Lustre: lustre-MDT0001: new disk, initializing [ 3312.617306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3312.643945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3312.654311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3315.365278] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3320.637810] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3323.355983] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3328.060250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3328.254658] Lustre: lustre-OST0000: new disk, initializing [ 3328.257878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3328.262340] Lustre: 113854:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3332.240503] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3337.334531] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3338.292779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3338.299122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3338.369738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3340.024575] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3340.151976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3342.631467] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3342.779308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3348.450562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3348.460429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3348.476581] Lustre: Skipped 1 previous similar message [ 3348.482053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3350.182900] Lustre: server umount lustre-OST0000 complete [ 3353.569994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3353.573750] LustreError: Skipped 1 previous similar message [ 3353.576921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.584366] Lustre: Skipped 1 previous similar message [ 3359.202303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3359.209078] Lustre: Skipped 5 previous similar messages [ 3359.455348] Lustre: server umount lustre-MDT0000 complete [ 3361.922251] LustreError: 112106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690634 with bad export cookie 17154421988513517557 [ 3361.923198] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3361.928773] LustreError: 112106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3362.135439] Lustre: server umount lustre-MDT0001 complete [ 3368.222955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.747878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.750909] Lustre: Skipped 1 previous similar message [ 3371.979250] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3374.051355] LustreError: 115924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3374.063595] LustreError: 115924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3377.591894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3380.420791] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3383.952625] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3386.342708] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3390.811452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3394.832423] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3398.483728] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3406.525869] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3409.889910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3409.894580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.903642] Lustre: Skipped 1 previous similar message [ 3409.906987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3414.589643] Lustre: server umount lustre-OST0000 complete [ 3422.996394] Lustre: server umount lustre-MDT0000 complete [ 3423.205154] LustreError: 116635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3423.217393] LustreError: 116635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3425.651937] LustreError: 115905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690697 with bad export cookie 17154421988513518439 [ 3425.656609] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3425.662798] LustreError: 115905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3425.960627] Lustre: server umount lustre-MDT0001 complete [ 3431.871758] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 3433.762433] Key type lgssc unregistered [ 3433.979593] LNet: 119193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3433.984615] LNetError: 119193:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3433.996952] LNet: Removed LNI 192.168.203.103@tcp [ 3434.600705] Key type .llcrypt unregistered [ 3434.602669] Key type ._llcrypt unregistered [ 3445.435938] Key type ._llcrypt registered [ 3445.438799] Key type .llcrypt registered [ 3445.513579] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3454.605864] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3455.263465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3455.335456] alg: No test for adler32 (adler32-zlib) [ 3456.282464] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 3456.452662] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 3458.095245] Key type lgssc registered [ 3458.717838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3475.945596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3475.953436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3477.092652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3477.116067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3477.166543] Lustre: lustre-MDT0000: new disk, initializing [ 3477.219554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3477.232972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3479.265703] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3485.439269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3485.516149] Lustre: 122287:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3485.556284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3485.559326] Lustre: Skipped 1 previous similar message [ 3485.609115] Lustre: lustre-MDT0001: new disk, initializing [ 3485.685191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3485.721612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3485.735821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3488.016601] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3492.595534] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3494.707629] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3498.495598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3498.701207] Lustre: lustre-OST0000: new disk, initializing [ 3498.705250] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3498.709078] Lustre: 124027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3498.781415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3499.076743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3499.086504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3499.150347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3501.915651] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3506.002032] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3508.034858] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3508.135530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3510.215920] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3510.339667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3514.849495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3514.851288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3514.853904] LustreError: Skipped 1 previous similar message [ 3514.861830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3514.867020] Lustre: Skipped 1 previous similar message [ 3517.503399] Lustre: server umount lustre-OST0000 complete [ 3519.970726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3519.976042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3519.986995] Lustre: Skipped 2 previous similar messages [ 3525.089329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3525.093052] Lustre: Skipped 2 previous similar messages [ 3530.208069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3530.217141] Lustre: Skipped 1 previous similar message [ 3534.305779] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3534.421285] Lustre: server umount lustre-MDT0000 complete [ 3536.530493] LustreError: 122277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690808 with bad export cookie 7407084444968549974 [ 3536.535324] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3536.537907] LustreError: 122277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3536.725784] Lustre: server umount lustre-MDT0001 complete [ 3541.335838] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:33:32 (1779690812) [ 3545.045193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.351839] LustreError: 126290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3545.365152] LustreError: 126290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3545.406540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3547.585639] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3550.688897] LustreError: 126291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3552.498235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3552.704400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3555.122375] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3558.448227] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3560.605368] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3562.913392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3562.914500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3562.923905] Lustre: Skipped 1 previous similar message [ 3573.216651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3573.223105] Lustre: Skipped 4 previous similar messages [ 3576.287158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3576.434807] Lustre: server umount lustre-MDT0000 complete [ 3578.336846] LustreError: 126290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.361878] LustreError: 126290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3578.570826] LustreError: 126272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690850 with bad export cookie 7407084444968550856 [ 3578.574440] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3578.580903] LustreError: 126272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3578.764926] Lustre: server umount lustre-MDT0001 complete [ 3583.405352] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:34:14 (1779690854) [ 3587.086717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3602.591644] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3607.712167] LustreError: 128476:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.103@tcp: failed processing log, type 4: rc = -110 [ 3633.375298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3636.816482] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3640.222360] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3641.397326] Lustre: server umount lustre-OST0000 complete [ 3645.914358] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:35:17 (1779690917) [ 3649.408248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.662217] LustreError: 129808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3649.718236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3652.387388] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3655.142488] LustreError: 129809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3658.216627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3661.437541] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3664.755334] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3666.630168] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3670.559802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3673.960070] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3676.861250] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3688.417656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3688.419248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3688.432970] Lustre: Skipped 2 previous similar messages [ 3688.439199] Lustre: Skipped 1 previous similar message [ 3691.972650] Lustre: server umount lustre-MDT0000 complete [ 3693.542104] LustreError: 130518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3693.552509] LustreError: 130518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3694.126364] LustreError: 129790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690966 with bad export cookie 7407084444968551507 [ 3694.132362] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3694.138904] LustreError: 129790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3694.332316] Lustre: server umount lustre-MDT0001 complete [ 3706.806726] Lustre: server umount lustre-OST0000 complete [ 3711.002828] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:36:22 (1779690982) [ 3714.434105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3714.673043] LustreError: 133118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3714.726537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.729052] Lustre: Skipped 2 previous similar messages [ 3717.113949] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3722.016894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3725.034155] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3728.606428] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3730.978296] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3736.736738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3739.070270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3744.529686] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3751.451712] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3758.223693] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3758.541925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3764.454133] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3764.628539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3767.776921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3767.783298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3767.784200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3767.805694] Lustre: Skipped 1 previous similar message [ 3767.818680] Lustre: Skipped 1 previous similar message [ 3781.087184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3781.204325] Lustre: server umount lustre-OST0000 complete [ 3788.256496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.051337] Lustre: server umount lustre-MDT0000 complete [ 3793.381333] LustreError: 133119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3793.399486] LustreError: 133119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3794.748300] LustreError: 133099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691066 with bad export cookie 7407084444968552564 [ 3794.757980] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3794.766729] LustreError: 133099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3795.182524] Lustre: server umount lustre-MDT0001 complete [ 3803.890158] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:37:54 (1779691074) [ 3809.542394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3825.056312] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3830.175498] LustreError: 136904:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.103@tcp: failed processing log, type 4: rc = -110 [ 3855.839455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3855.846635] Lustre: Skipped 2 previous similar messages [ 3860.289904] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3864.219774] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3868.577462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3868.831930] LustreError: 136929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3868.948487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3871.877541] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3877.599897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3881.227446] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3886.165130] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3889.249606] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3892.658188] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3892.851655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3895.895283] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3896.057863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3898.336247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3898.341591] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3898.347983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3898.356893] Lustre: Skipped 3 previous similar messages [ 3898.365117] Lustre: Skipped 8 previous similar messages [ 3903.637799] Lustre: server umount lustre-MDT0000 complete [ 3906.540243] LustreError: 136911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691178 with bad export cookie 7407084444968553600 [ 3906.543721] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3906.552628] LustreError: 136911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3906.794900] Lustre: server umount lustre-MDT0001 complete [ 3920.254858] Lustre: server umount lustre-OST0000 complete [ 3926.575596] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:39:57 (1779691197) [ 3932.079468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3947.615599] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3952.735545] LustreError: 140721:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.103@tcp: failed processing log, type 4: rc = -110 [ 3983.317429] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3988.024662] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3993.316369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3993.715231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3993.719510] Lustre: Skipped 3 previous similar messages [ 3993.793623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3997.210565] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4000.230629] LustreError: 141647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4000.240538] LustreError: 141647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4003.666568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4006.933279] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4011.318693] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4014.090634] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4019.057846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4019.166947] Lustre: lustre-OST0001: new disk, initializing [ 4019.175974] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4019.180347] Lustre: 143254:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4023.530368] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4027.422568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4027.432247] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4027.519353] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4029.153882] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4032.682396] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4032.828879] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4035.748713] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4035.912571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4037.607920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.611897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4037.622274] Lustre: Skipped 1 previous similar message [ 4037.631630] Lustre: Skipped 3 previous similar messages [ 4043.449774] Lustre: server umount lustre-OST0000 complete [ 4047.846452] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4060.128961] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4060.215735] Lustre: server umount lustre-OST0001 complete [ 4068.867565] Lustre: server umount lustre-MDT0000 complete [ 4071.635935] LustreError: 143253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691343 with bad export cookie 7407084444968554566 [ 4071.636990] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4071.646172] LustreError: 143253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4071.882954] Lustre: server umount lustre-MDT0001 complete [ 4085.352803] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:42:36 (1779691356) [ 4086.277677] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4087.204540] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4088.040678] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:42:39 (1779691359) [ 4091.860953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4092.035678] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4092.054947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4094.789992] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4100.086857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.150492] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4100.165741] Lustre: 146662:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4102.515463] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4105.903782] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4108.108692] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4112.184648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4112.336420] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4116.954308] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4118.594084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4120.531356] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4122.957244] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4123.102122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4125.780408] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4125.938607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4128.739336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4128.741824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4128.746761] LustreError: Skipped 1 previous similar message [ 4128.764312] Lustre: Skipped 5 previous similar messages [ 4133.419445] Lustre: server umount lustre-OST0000 complete [ 4150.358193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4153.951980] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4157.087464] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4160.079805] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4168.935857] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4168.949322] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4168.950331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4169.565197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 4171.904399] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4172.037664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4174.254938] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8ceb11bbd000.ost_server_uuid 50 [ 4175.088905] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8ceb11bbd000.ost_server_uuid in FULL state after 0 sec [ 4175.184213] LustreError: 146671:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.3@tcp) returned error from blocking AST (req@0000000023d0f83d x1866140985029376 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000007570855/0x66cb400eebac19a0 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.3@tcp remote: 0xa220b0fb5f44a993 expref: 5 pid: 146671 timeout: 4275 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4175.211380] LustreError: lustre-MDT0000: A client on nid 192.168.203.3@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4175.217909] LustreError: 146660:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.3@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000007570855/0x66cb400eebac19a0 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.3@tcp remote: 0xa220b0fb5f44a993 expref: 6 pid: 146671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4179.427958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4183.591593] Lustre: server umount lustre-OST0000 complete [ 4191.982346] Lustre: server umount lustre-MDT0000 complete [ 4194.147295] LustreError: 146655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691466 with bad export cookie 7407084444968555749 [ 4194.150582] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4194.156821] LustreError: 146655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4200.466214] Lustre: server umount lustre-MDT0001 complete [ 4206.014786] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4207.521658] Key type lgssc unregistered [ 4207.741797] LNet: 152207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4207.755058] LNetError: 152207:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4207.770072] LNet: Removed LNI 192.168.203.103@tcp [ 4208.308416] Key type .llcrypt unregistered [ 4208.310333] Key type ._llcrypt unregistered [ 4221.847399] Key type ._llcrypt registered [ 4221.849596] Key type .llcrypt registered [ 4221.918348] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4222.372022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4222.425143] alg: No test for adler32 (adler32-zlib) [ 4223.354725] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 4223.492439] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4225.103178] Key type lgssc registered [ 4225.744605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4230.207138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4230.220893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4231.517263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4233.931271] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4239.114378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4239.349381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4241.600483] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4244.871888] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4247.040294] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4251.300614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4251.526603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4254.750826] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4256.695341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4257.989701] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4266.685286] Lustre: Failing over lustre-MDT0000 [ 4266.850988] Lustre: server umount lustre-MDT0000 complete [ 4269.026262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.027458] LustreError: 153574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.034277] Lustre: Skipped 1 previous similar message [ 4269.047255] LustreError: 153574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4274.149160] LustreError: 153580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.160719] LustreError: 153580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4278.204404] LustreError: 153561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.3@tcp arrived at 1779691550 with bad export cookie 12492193355986674721 [ 4278.217193] LustreError: 153561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4279.266419] LustreError: 153579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.277695] LustreError: 153579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4282.102127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4282.181075] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4282.329362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4282.364522] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4284.495294] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4285.772450] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4285.777990] Lustre: lustre-MDT0000: Denying connection for new client d32b20ad-1388-4943-bb2a-eab789ef288d (at 192.168.203.3@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 4287.467321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4294.625334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4294.631619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.641789] Lustre: Skipped 1 previous similar message [ 4294.646516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4297.699082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4297.701258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.001388] Lustre: server umount lustre-OST0000 complete [ 4300.931464] LustreError: 156948:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4300.935917] Lustre: 156314:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4300.940772] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4300.952773] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4300.958134] LustreError: 156314:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e7dfe7fd x1866141789098496/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4300.968160] LustreError: 156314:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4300.973067] LustreError: 156314:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4300.977849] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 4301.103422] Lustre: server umount lustre-MDT0000 complete [ 4302.817443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4302.825044] LustreError: 155190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4302.998474] LustreError: 153561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691575 with bad export cookie 12492193355986674826 [ 4303.001837] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4303.003052] LustreError: 153561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4303.004258] LustreError: 152890:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4309.222775] Lustre: server umount lustre-MDT0001 complete [ 4314.442943] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4315.914395] Key type lgssc unregistered [ 4316.093711] LNet: 157684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4316.101815] LNetError: 157684:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4316.114404] LNet: Removed LNI 192.168.203.103@tcp [ 4316.593123] Key type .llcrypt unregistered [ 4316.595267] Key type ._llcrypt unregistered [ 4328.557823] Key type ._llcrypt registered [ 4328.559532] Key type .llcrypt registered [ 4328.621485] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4329.191857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4329.202343] alg: No test for adler32 (adler32-zlib) [ 4330.089271] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 4330.189645] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4331.799162] Key type lgssc registered [ 4332.288607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4335.770169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4335.781401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.004536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4338.502623] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4342.224305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4342.430163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4344.147663] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4346.690229] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4348.369849] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4351.683438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4351.883402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.455529] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4357.047243] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4362.169388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4367.332078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4367.333577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4367.334467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4367.340112] Lustre: Skipped 1 previous similar message [ 4371.109480] Lustre: server umount lustre-OST0000 complete [ 4373.471952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4373.475594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.481600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4373.483988] Lustre: Skipped 1 previous similar message [ 4377.569585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4377.571742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4382.688396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4382.691326] Lustre: Skipped 2 previous similar messages [ 4387.295302] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4387.371257] Lustre: server umount lustre-MDT0000 complete [ 4387.808679] LustreError: 159055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4387.815400] LustreError: 159055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4388.959908] LustreError: 159038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691661 with bad export cookie 16604780039700370086 [ 4388.963811] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4388.966967] LustreError: 159038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4389.121590] Lustre: server umount lustre-MDT0001 complete [ 4393.210569] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4394.418037] Key type lgssc unregistered [ 4394.567630] LNet: 162229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4394.571295] LNetError: 162229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4394.580519] LNet: Removed LNI 192.168.203.103@tcp [ 4394.917921] Key type .llcrypt unregistered [ 4394.919636] Key type ._llcrypt unregistered [ 4408.671785] Key type ._llcrypt registered [ 4408.673581] Key type .llcrypt registered [ 4408.736723] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4409.202425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4409.254204] alg: No test for adler32 (adler32-zlib) [ 4410.138156] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 4410.241747] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4411.847164] Key type lgssc registered [ 4412.325517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4415.441039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4415.450523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4416.682727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4418.433427] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4422.206815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4422.403567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4424.102605] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4426.143951] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4427.612931] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4430.248916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4430.410933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.598081] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4434.645274] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4441.703983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4447.202895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4447.203193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4447.209180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4447.223478] Lustre: Skipped 1 previous similar message [ 4452.322991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4452.326567] Lustre: Skipped 2 previous similar messages [ 4453.152553] Lustre: server umount lustre-OST0000 complete [ 4457.440663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.441534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.446119] Lustre: Skipped 1 previous similar message [ 4462.560070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4462.562987] Lustre: Skipped 2 previous similar messages [ 4467.680261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4467.686255] Lustre: Skipped 1 previous similar message [ 4469.215129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4469.297607] Lustre: server umount lustre-MDT0000 complete [ 4471.069149] LustreError: 166351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691743 with bad export cookie 13607691322276927418 [ 4471.073074] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4471.073760] LustreError: 166351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4471.242550] Lustre: server umount lustre-MDT0001 complete [ 4475.675770] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4476.856458] Key type lgssc unregistered [ 4477.000763] LNet: 167086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4477.006305] LNetError: 167086:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4477.015581] LNet: Removed LNI 192.168.203.103@tcp [ 4477.372721] Key type .llcrypt unregistered [ 4477.375090] Key type ._llcrypt unregistered [ 4487.162265] Key type ._llcrypt registered [ 4487.163711] Key type .llcrypt registered [ 4487.209255] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4487.616703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4487.669545] alg: No test for adler32 (adler32-zlib) [ 4488.549407] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 4488.654528] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4490.239333] Key type lgssc registered [ 4490.704106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4495.368174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4495.381606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4496.571685] Lustre: *** cfs_fail_loc=135, val=0*** [ 4496.616763] LustreError: 168569:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 4496.620264] LustreError: 168569:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.103@tcp: cfg command failed: rc = -2 [ 4496.624849] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4496.624849] [ 4496.629649] LustreError: MGC192.168.203.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4496.636417] LustreError: 168530:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4496.641755] LustreError: 168530:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 4496.645701] LustreError: 168530:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 4496.650912] LustreError: 168530:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 4496.703609] Lustre: server umount lustre-MDT0000 complete [ 4496.705821] LustreError: 168530:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4499.947046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.179063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4501.979505] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4505.591148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.797642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4507.679697] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4510.137125] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4511.729522] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4515.298410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.301355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4515.304831] Lustre: Skipped 1 previous similar message [ 4515.314369] Lustre: Skipped 1 previous similar message [ 4516.322704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4516.326768] Lustre: Skipped 1 previous similar message [ 4518.816895] Lustre: server umount lustre-MDT0000 complete [ 4520.256261] LustreError: 170280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691792 with bad export cookie 17107276033987094816 [ 4520.258662] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4520.261570] LustreError: 170280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4520.410023] Lustre: server umount lustre-MDT0001 complete [ 4524.498666] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4525.686144] Key type lgssc unregistered [ 4525.821469] LNet: 171015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4525.824573] LNetError: 171015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4525.834357] LNet: Removed LNI 192.168.203.103@tcp [ 4526.171687] Key type .llcrypt unregistered [ 4526.173199] Key type ._llcrypt unregistered [ 4535.103437] Key type ._llcrypt registered [ 4535.105065] Key type .llcrypt registered [ 4535.147901] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4535.270581] Key type .llcrypt unregistered [ 4535.273100] Key type ._llcrypt unregistered [ 4543.135340] Key type ._llcrypt registered [ 4543.136584] Key type .llcrypt registered [ 4543.187097] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4543.583802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4543.612060] alg: No test for adler32 (adler32-zlib) [ 4544.501470] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 4544.606932] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4546.207653] Key type lgssc registered [ 4546.780201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4550.244115] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4550.255762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4565.727311] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4570.847836] LustreError: 173088:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.103@tcp: failed processing log, type 4: rc = -110 [ 4597.471178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4600.587825] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4603.285758] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4606.314272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4606.554075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4606.585494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4608.215095] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4611.553176] LustreError: 173114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4611.560418] LustreError: 173114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4611.730698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4611.916938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4613.524889] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4615.884545] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4617.436970] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4618.739484] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4621.793734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4621.798606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.808771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4622.304609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4622.308653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.317474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4623.333085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4623.336890] Lustre: Skipped 1 previous similar message [ 4627.604268] Lustre: server umount lustre-OST0000 complete [ 4632.545518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4632.549397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.556625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4635.752831] Lustre: server umount lustre-MDT0000 complete [ 4637.648422] LustreError: 175409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691909 with bad export cookie 5487913433942676717 [ 4637.650034] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4637.657220] LustreError: 175409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4637.799154] Lustre: server umount lustre-MDT0001 complete [ 4642.431310] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4643.832826] Key type lgssc unregistered [ 4643.992138] LNet: 176548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4643.996510] LNetError: 176548:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4644.009469] LNet: Removed LNI 192.168.203.103@tcp [ 4644.391172] Key type .llcrypt unregistered [ 4644.392916] Key type ._llcrypt unregistered [ 4655.206470] Key type ._llcrypt registered [ 4655.208312] Key type .llcrypt registered [ 4655.254982] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4655.717946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4655.744671] alg: No test for adler32 (adler32-zlib) [ 4656.634289] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 4656.739522] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4658.327161] Key type lgssc registered [ 4658.794533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4661.931895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4661.943867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4663.191621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.852699] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4668.477664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.641345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4670.173802] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4672.353328] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4673.835573] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4676.691432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4676.884739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4679.308404] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4681.454507] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4688.179726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4694.948856] Lustre: Failing over lustre-MDT0000 [ 4695.073972] Lustre: server umount lustre-MDT0000 complete [ 4698.244376] LustreError: 179538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4698.256082] LustreError: 179538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4698.595416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4698.602876] Lustre: Skipped 2 previous similar messages [ 4703.347092] LustreError: 179661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4703.357346] LustreError: 179661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4708.466688] LustreError: 177917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.480348] LustreError: 177917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4709.194073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4709.270933] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4709.394378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4709.421887] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4711.116183] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4713.369057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4713.585819] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4714.471651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4714.481631] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4714.500977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4715.707949] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4724.192344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4724.195459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4724.201853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4724.706876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4728.423059] Lustre: server umount lustre-OST0000 complete [ 4734.944850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4734.944852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4734.944862] Lustre: Skipped 1 previous similar message [ 4734.945675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4736.283475] Lustre: server umount lustre-MDT0000 complete [ 4737.840272] LustreError: 179539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692010 with bad export cookie 11120345881866444302 [ 4737.843469] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4737.845533] LustreError: 179539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4737.971762] Lustre: server umount lustre-MDT0001 complete [ 4742.248717] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4743.417390] Key type lgssc unregistered [ 4743.564706] LNet: 182352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4743.569504] LNetError: 182352:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4743.578493] LNet: Removed LNI 192.168.203.103@tcp [ 4743.940104] Key type .llcrypt unregistered [ 4743.942233] Key type ._llcrypt unregistered [ 4754.600799] Key type ._llcrypt registered [ 4754.602252] Key type .llcrypt registered [ 4754.654209] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4755.050212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4755.104080] alg: No test for adler32 (adler32-zlib) [ 4756.004936] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 4756.121107] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4757.719225] Key type lgssc registered [ 4758.243257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4761.528438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4761.538503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4762.735475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4764.246072] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4767.430420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4767.598894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4769.141721] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4771.268269] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4772.733561] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4775.555996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4775.728148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4777.923407] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4780.372892] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4786.982675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4792.761598] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4799.485784] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4808.833602] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4817.892339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4817.892604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4817.894331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4817.904208] Lustre: Skipped 1 previous similar message [ 4823.008730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4823.013644] Lustre: Skipped 2 previous similar messages [ 4823.450325] Lustre: server umount lustre-OST0000 complete [ 4828.129565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4828.130705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4828.136062] Lustre: Skipped 1 previous similar message [ 4833.248414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4833.251481] Lustre: Skipped 2 previous similar messages [ 4838.368521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4838.371196] Lustre: Skipped 1 previous similar message [ 4839.391112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4839.454796] Lustre: server umount lustre-MDT0000 complete [ 4840.870183] LustreError: 183704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692113 with bad export cookie 8879845162048666081 [ 4840.873383] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4840.879062] LustreError: 183704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4841.006672] Lustre: server umount lustre-MDT0001 complete [ 4845.104447] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4846.341302] Key type lgssc unregistered [ 4846.490640] LNet: 187089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4846.493934] LNetError: 187089:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4846.501560] LNet: Removed LNI 192.168.203.103@tcp [ 4846.836240] Key type .llcrypt unregistered [ 4846.837699] Key type ._llcrypt unregistered [ 4856.429599] Key type ._llcrypt registered [ 4856.431099] Key type .llcrypt registered [ 4856.475885] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4856.900264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4856.914913] alg: No test for adler32 (adler32-zlib) [ 4857.773499] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 4857.861827] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4859.455236] Key type lgssc registered [ 4859.875377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4862.867635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4862.875087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.053912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4865.548570] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4868.908168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4869.054467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4870.546043] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4872.593768] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4874.074523] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4876.838384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4877.004609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4879.322341] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4881.680030] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4889.317463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4895.304241] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4905.118597] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4913.786987] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4920.886770] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4930.533100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4930.533516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.545323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4930.548400] Lustre: Skipped 1 previous similar message [ 4935.650563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4935.655995] Lustre: Skipped 1 previous similar message [ 4936.679158] Lustre: server umount lustre-OST0000 complete [ 4940.768781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.773546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4940.773687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4940.773691] LustreError: Skipped 1 previous similar message [ 4940.781642] Lustre: Skipped 2 previous similar messages [ 4940.800717] Lustre: Skipped 1 previous similar message [ 4945.891024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4945.896087] Lustre: Skipped 1 previous similar message [ 4946.391436] Lustre: server umount lustre-MDT0000 complete [ 4949.319870] LustreError: 191310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692221 with bad export cookie 4434943220450482115 [ 4949.320631] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4949.329529] LustreError: 191310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4949.557596] Lustre: server umount lustre-MDT0001 complete [ 4959.670874] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4962.104676] Key type lgssc unregistered [ 4962.298219] LNet: 193479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4962.303867] LNetError: 193479:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4963.365836] LNet: Removed LNI 192.168.203.103@tcp [ 4963.970612] Key type .llcrypt unregistered [ 4963.974048] Key type ._llcrypt unregistered [ 4985.985625] Key type ._llcrypt registered [ 4985.988933] Key type .llcrypt registered [ 4986.105532] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4987.169093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4987.339843] alg: No test for adler32 (adler32-zlib) [ 4988.511281] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 4988.719714] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4990.407180] Key type lgssc registered [ 4991.660045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4998.234830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4998.256118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4999.814444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5002.963692] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5009.384523] LustreError: 194856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5009.398023] LustreError: 194856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5010.220150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5010.555048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5013.904667] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5018.481206] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5021.725138] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5026.808479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5027.099961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5031.857439] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5035.792351] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5044.522342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5054.946074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5054.954596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5054.962220] Lustre: Skipped 1 previous similar message [ 5054.968453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5060.069635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5060.072651] Lustre: Skipped 2 previous similar messages [ 5061.006275] Lustre: server umount lustre-OST0000 complete [ 5065.184577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5065.193730] Lustre: Skipped 1 previous similar message [ 5065.194489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5065.199687] Lustre: Skipped 1 previous similar message [ 5069.711992] Lustre: server umount lustre-MDT0000 complete [ 5070.304232] LustreError: 194852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5070.316954] LustreError: 194852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5072.308804] LustreError: 194836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692344 with bad export cookie 2875169130344277357 [ 5072.310145] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5072.316774] LustreError: 194836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5072.516709] Lustre: server umount lustre-MDT0001 complete [ 5079.449848] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5081.499196] Key type lgssc unregistered [ 5081.754649] LNet: 198200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5081.765343] LNetError: 198200:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5081.782423] LNet: Removed LNI 192.168.203.103@tcp [ 5082.399610] Key type .llcrypt unregistered [ 5082.402930] Key type ._llcrypt unregistered [ 5095.654193] Key type ._llcrypt registered [ 5095.657775] Key type .llcrypt registered [ 5095.738539] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5096.556746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5096.609018] alg: No test for adler32 (adler32-zlib) [ 5097.635793] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 5097.811433] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5099.455161] Key type lgssc registered [ 5100.249546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5105.611848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5105.635368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5106.928693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5109.935668] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5115.284410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.544843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5118.284879] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5121.807550] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5124.241663] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5128.852047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5129.122334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5132.201619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5132.850238] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5137.102646] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5147.618729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5147.627196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.638451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5150.691525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5150.694168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5150.698786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.721614] Lustre: Skipped 1 previous similar message [ 5153.217273] Lustre: server umount lustre-OST0000 complete [ 5156.320370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5156.327574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5156.342575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5160.928847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5160.930487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5166.051728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5166.057085] Lustre: Skipped 2 previous similar messages [ 5170.143121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5170.249940] Lustre: server umount lustre-MDT0000 complete [ 5172.444284] LustreError: 199407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692444 with bad export cookie 2849761022102368865 [ 5172.450080] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5172.455700] LustreError: 199407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5172.653412] Lustre: server umount lustre-MDT0001 complete [ 5178.043735] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5179.382679] Key type lgssc unregistered [ 5179.537546] LNet: 202750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5179.544342] LNetError: 202750:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5179.556900] LNet: Removed LNI 192.168.203.103@tcp [ 5179.979674] Key type .llcrypt unregistered [ 5179.981645] Key type ._llcrypt unregistered [ 5190.149769] Key type ._llcrypt registered [ 5190.151468] Key type .llcrypt registered [ 5190.201848] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5190.785673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5190.798093] alg: No test for adler32 (adler32-zlib) [ 5191.804231] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 5191.936103] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5193.551167] Key type lgssc registered [ 5194.156944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5198.519988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5198.534948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.873494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5202.259823] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5207.299729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5207.536361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5209.670609] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5212.985501] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5215.119361] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5218.912854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5219.118161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5222.363621] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5225.260835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5225.852650] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5233.083906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5233.132906] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5233.215288] Lustre: lustre-OST0001: Not available for connect from 192.168.203.3@tcp (not set up) [ 5233.316543] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5237.445220] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5241.132987] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5242.150360] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5253.245198] Lustre: Permanently deactivating lustre-OST0001 [ 5253.262610] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5258.776029] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5263.344472] Lustre: Permanently reactivating lustre-OST0001 [ 5263.353529] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5263.357026] Lustre: Skipped 2 previous similar messages [ 5265.227430] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.235811] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5265.241934] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5265.249852] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5265.269194] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5275.616758] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5275.618087] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.628165] LustreError: Skipped 1 previous similar message [ 5275.641519] Lustre: Skipped 1 previous similar message [ 5275.642279] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5275.654182] Lustre: Skipped 1 previous similar message [ 5280.231525] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5280.236827] Lustre: Skipped 1 previous similar message [ 5281.561690] Lustre: server umount lustre-OST0001 complete [ 5285.348090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.348878] LustreError: 206497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.361553] Lustre: Skipped 1 previous similar message [ 5285.380837] LustreError: 206497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5285.388309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5285.391101] Lustre: Skipped 1 previous similar message [ 5290.467099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5290.468466] LustreError: 205588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.475057] Lustre: Skipped 1 previous similar message [ 5290.489309] LustreError: 205588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5295.587704] LustreError: 206497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5295.601891] LustreError: 206497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5298.143087] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5298.239192] Lustre: server umount lustre-OST0000 complete [ 5300.706522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5300.712544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5300.719181] Lustre: Skipped 2 previous similar messages [ 5300.726662] Lustre: Skipped 3 previous similar messages [ 5306.771237] Lustre: server umount lustre-MDT0000 complete [ 5308.768099] LustreError: 203975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692580 with bad export cookie 4041757098491633277 [ 5308.769551] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5308.778121] LustreError: 203975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5308.998530] Lustre: server umount lustre-MDT0001 complete [ 5315.499231] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5317.553234] Key type lgssc unregistered [ 5317.717295] LNet: 208594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5317.724227] LNetError: 208594:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5317.740883] LNet: Removed LNI 192.168.203.103@tcp [ 5318.295167] Key type .llcrypt unregistered [ 5318.297346] Key type ._llcrypt unregistered [ 5340.174292] Key type ._llcrypt registered [ 5340.176232] Key type .llcrypt registered [ 5340.262549] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5340.843386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5340.863270] alg: No test for adler32 (adler32-zlib) [ 5341.791830] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 5341.935465] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5343.567735] Key type lgssc registered [ 5344.215703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5348.548748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5348.565350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5349.720943] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5349.737669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5349.833737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5352.271602] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5356.985661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5357.051129] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5357.069625] Lustre: 210758:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5357.228168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5359.525952] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5362.727421] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5364.726217] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5368.234077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5368.327977] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5368.431057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5371.525462] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5371.753408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 5374.262929] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5376.201384] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5384.054603] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5392.074666] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5399.136406] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5408.182712] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5429.088940] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5429.093611] Lustre: Skipped 2 previous similar messages [ 5466.917250] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5466.921570] Lustre: Skipped 4 previous similar messages [ 5531.852298] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 5531.856476] Lustre: Skipped 8 previous similar messages [ 5535.717765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5535.720402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5535.724183] Lustre: Skipped 1 previous similar message [ 5535.731296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5535.734568] Lustre: Skipped 1 previous similar message [ 5540.139637] Lustre: server umount lustre-OST0000 complete [ 5545.953148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.956040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5545.960354] Lustre: Skipped 1 previous similar message [ 5548.156081] Lustre: server umount lustre-MDT0000 complete [ 5550.134457] LustreError: 210749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692822 with bad export cookie 5251930186415609637 [ 5550.137409] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5550.142016] LustreError: 210749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5550.324478] Lustre: server umount lustre-MDT0001 complete [ 5556.121528] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5557.670897] Key type lgssc unregistered [ 5557.847097] LNet: 215187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5557.852991] LNetError: 215187:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5557.862714] LNet: Removed LNI 192.168.203.103@tcp [ 5558.278646] Key type .llcrypt unregistered [ 5558.280344] Key type ._llcrypt unregistered [ 5569.808616] Key type ._llcrypt registered [ 5569.810490] Key type .llcrypt registered [ 5569.870700] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5570.297590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5570.360106] alg: No test for adler32 (adler32-zlib) [ 5571.267851] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 5571.391944] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5573.007502] Key type lgssc registered [ 5573.625627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5577.533703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5577.544303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5578.796321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5580.823809] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5585.423377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5585.695049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5587.987415] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5590.749506] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5592.616790] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5596.043502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5596.249215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.014760] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5601.723909] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5603.431919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 5614.048749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5614.056755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5614.064490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5616.099298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5616.099645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5616.105984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.872454] Lustre: server umount lustre-OST0000 complete [ 5621.728288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5621.733912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5621.742642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5621.746054] Lustre: Skipped 1 previous similar message [ 5626.336483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5626.336749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.806551] Lustre: server umount lustre-MDT0000 complete [ 5629.624801] LustreError: 218800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692901 with bad export cookie 2693949312638873897 [ 5629.627446] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5629.631618] LustreError: 218800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5629.782825] Lustre: server umount lustre-MDT0001 complete [ 5635.260061] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5637.106881] Key type lgssc unregistered [ 5637.308405] LNet: 219891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5637.312697] LNetError: 219891:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5637.323864] LNet: Removed LNI 192.168.203.103@tcp [ 5637.957417] Key type .llcrypt unregistered [ 5637.959870] Key type ._llcrypt unregistered [ 5670.157435] Key type ._llcrypt registered [ 5670.163615] Key type .llcrypt registered [ 5670.391144] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5672.438332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5672.517587] alg: No test for adler32 (adler32-zlib) [ 5674.330167] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 5674.987655] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5676.799504] Key type lgssc registered [ 5680.129487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5951.163472] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5954.324520] Key type lgssc unregistered [ 5954.612747] LNet: 221881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5954.623880] LNetError: 221881:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5954.638877] LNet: Removed LNI 192.168.203.103@tcp [ 5955.399484] Key type .llcrypt unregistered [ 5955.409424] Key type ._llcrypt unregistered [ 5983.809953] Key type ._llcrypt registered [ 5983.812678] Key type .llcrypt registered [ 5983.903732] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 6003.774378] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 6005.126960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6005.146176] alg: No test for adler32 (adler32-zlib) [ 6006.406350] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 6006.840919] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 6008.528119] Key type lgssc registered [ 6010.163483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6047.282125] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5734 sec ========== 03:15:17 (1779693317) [ 6049.074875] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:15:19 (1779693319) === [ 6052.602225] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:15:23 (1779693323) === [ 6096.024554] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 6098.635560] Key type lgssc unregistered [ 6098.929721] LNet: 227071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6098.941769] LNetError: 227071:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6098.968614] LNet: Removed LNI 192.168.203.103@tcp [ 6099.767658] Key type .llcrypt unregistered [ 6099.770055] Key type ._llcrypt unregistered