[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 472494437 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2464MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002008] Switched APIC routing to physical x2apic. [ 0.003014] kvm-guest: setup PV IPIs [ 0.005493] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007009] pid_max: default: 32768 minimum: 301 [ 0.008128] LSM: Security Framework initializing [ 0.009041] Yama: becoming mindful. [ 0.010020] SELinux: Initializing. [ 0.011038] *** VALIDATE selinux *** [ 0.017574] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021448] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022108] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023077] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024098] *** VALIDATE tmpfs *** [ 0.025000] *** VALIDATE proc *** [ 0.025034] *** VALIDATE cgroup *** [ 0.026007] *** VALIDATE cgroup2 *** [ 0.027246] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.028136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.029000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.029028] Spectre V2 : User space: Vulnerable [ 0.030004] Speculative Store Bypass: Vulnerable [ 0.031504] debug: unmapping init [mem 0xffffffffa5659000-0xffffffffa5660fff] [ 0.033000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.033689] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.034023] ... version: 2 [ 0.035007] ... bit width: 48 [ 0.035875] ... generic registers: 4 [ 0.036007] ... value mask: 0000ffffffffffff [ 0.036927] ... max period: 00007fffffffffff [ 0.037012] ... fixed-purpose events: 3 [ 0.038006] ... event mask: 000000070000000f [ 0.039242] rcu: Hierarchical SRCU implementation. [ 0.041236] smp: Bringing up secondary CPUs ... [ 0.042516] x86: Booting SMP configuration: [ 0.043022] .... node #0, CPUs: #1 #2 #3 [ 0.048113] smp: Brought up 1 node, 4 CPUs [ 0.050009] smpboot: Max logical packages: 1 [ 0.050938] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.094303] node 0 deferred pages initialised in 42ms [ 0.098161] devtmpfs: initialized [ 0.099304] x86/mm: Memory block size: 128MB [ 0.103687] gcov: version magic: 0x41383552 [ 0.105430] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.106134] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.107444] pinctrl core: initialized pinctrl subsystem [ 0.108291] [ 0.108749] ************************************************************* [ 0.109022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.110033] ** ** [ 0.111032] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.112023] ** ** [ 0.113026] ** This means that this kernel is built to expose internal ** [ 0.114028] ** IOMMU data structures, which may compromise security on ** [ 0.115022] ** your system. ** [ 0.116020] ** ** [ 0.117019] ** If you see this message and you are not debugging the ** [ 0.118018] ** kernel, report this immediately to your vendor! ** [ 0.119015] ** ** [ 0.120010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.121024] ************************************************************* [ 0.122756] NET: Registered protocol family 16 [ 0.123467] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.124071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.125076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.126568] cpuidle: using governor menu [ 0.129386] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.135355] PCI: Using configuration type 1 for base access [ 0.138822] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.155111] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.156000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.158260] cryptd: max_cpu_qlen set to 1000 [ 0.159399] ACPI: Added _OSI(Module Device) [ 0.160000] ACPI: Added _OSI(Processor Device) [ 0.161020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.163320] ACPI: Added _OSI(Processor Aggregator Device) [ 0.173174] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.182252] ACPI: Interpreter enabled [ 0.184047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.185022] ACPI: Using IOAPIC for interrupt routing [ 0.187158] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.190470] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.202258] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.204023] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.206012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.209069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.215900] acpiphp: Slot [2] registered [ 0.216098] acpiphp: Slot [5] registered [ 0.218098] acpiphp: Slot [6] registered [ 0.220103] acpiphp: Slot [7] registered [ 0.221112] acpiphp: Slot [8] registered [ 0.222159] acpiphp: Slot [9] registered [ 0.224103] acpiphp: Slot [10] registered [ 0.226098] acpiphp: Slot [3] registered [ 0.227089] acpiphp: Slot [4] registered [ 0.229125] acpiphp: Slot [11] registered [ 0.230110] acpiphp: Slot [12] registered [ 0.232089] acpiphp: Slot [13] registered [ 0.233105] acpiphp: Slot [14] registered [ 0.235088] acpiphp: Slot [15] registered [ 0.236067] acpiphp: Slot [16] registered [ 0.237158] acpiphp: Slot [17] registered [ 0.239068] acpiphp: Slot [18] registered [ 0.240054] acpiphp: Slot [19] registered [ 0.241076] acpiphp: Slot [20] registered [ 0.242052] acpiphp: Slot [21] registered [ 0.243054] acpiphp: Slot [22] registered [ 0.244057] acpiphp: Slot [23] registered [ 0.245057] acpiphp: Slot [24] registered [ 0.247066] acpiphp: Slot [25] registered [ 0.248153] acpiphp: Slot [26] registered [ 0.250107] acpiphp: Slot [27] registered [ 0.251070] acpiphp: Slot [28] registered [ 0.252056] acpiphp: Slot [29] registered [ 0.254071] acpiphp: Slot [30] registered [ 0.255078] acpiphp: Slot [31] registered [ 0.256056] PCI host bridge to bus 0000:00 [ 0.258048] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.260036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.262014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.264013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.265012] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.268075] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.270193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.271856] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.275352] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.284012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.287434] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.289009] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.290008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.291008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.293189] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.294735] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.297035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.299760] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.304014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.317955] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.321918] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.328000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.333022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.339020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.358013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.367163] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.371023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.376010] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.388023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.395000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.422015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.428023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.456014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.471257] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.477014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.482023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.508024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.528590] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.540019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.551038] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.578020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.591911] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.607026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.614013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.629017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.642567] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.645547] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.650323] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.653373] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.654152] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.655960] iommu: Default domain type: Passthrough [ 0.656408] SCSI subsystem initialized [ 0.657000] ACPI: bus type USB registered [ 0.658117] usbcore: registered new interface driver usbfs [ 0.660072] usbcore: registered new interface driver hub [ 0.661118] usbcore: registered new device driver usb [ 0.662191] pps_core: LinuxPPS API ver. 1 registered [ 0.663008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.666060] PTP clock support registered [ 0.670093] EDAC MC: Ver: 3.0.0 [ 0.671178] PCI: Using ACPI for IRQ routing [ 0.672374] NetLabel: Initializing [ 0.674008] NetLabel: domain hash size = 128 [ 0.675006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.676000] NetLabel: unlabeled traffic allowed by default [ 0.676003] vgaarb: loaded [ 0.677505] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.679011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.689916] clocksource: Switched to clocksource kvm-clock [ 0.909468] VFS: Disk quotas dquot_6.6.0 [ 0.910902] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.913223] *** VALIDATE ramfs *** [ 0.914158] *** VALIDATE hugetlbfs *** [ 0.915468] pnp: PnP ACPI init [ 0.919356] pnp: PnP ACPI: found 6 devices [ 0.937303] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.941337] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.944185] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.946442] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.948679] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.951020] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.953828] NET: Registered protocol family 2 [ 0.956892] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.961630] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.965709] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.980753] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.014497] TCP: Hash tables configured (established 65536 bind 65536) [ 1.032262] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.035597] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.038742] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.041933] NET: Registered protocol family 1 [ 1.044644] RPC: Registered named UNIX socket transport module. [ 1.047422] RPC: Registered udp transport module. [ 1.049541] RPC: Registered tcp transport module. [ 1.051477] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.053549] NET: Registered protocol family 44 [ 1.054584] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.056505] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.057958] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.059780] PCI: CLS 0 bytes, default 64 [ 1.060943] Unpacking initramfs... [ 3.396491] debug: unmapping init [mem 0xffff8abc3cc54000-0xffff8abc3ffbffff] [ 3.407740] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.417764] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 3.427730] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.482035] Initialise system trusted keyrings [ 4.484374] Key type blacklist registered [ 4.486931] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.503048] zbud: loaded [ 4.507859] *** VALIDATE nfs *** [ 4.508793] *** VALIDATE nfs4 *** [ 4.511137] pstore: using deflate compression [ 4.517205] Platform Keyring initialized [ 4.702688] NET: Registered protocol family 38 [ 4.704222] Key type asymmetric registered [ 4.705251] Asymmetric key parser 'x509' registered [ 4.706700] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.708803] io scheduler mq-deadline registered [ 4.710092] io scheduler kyber registered [ 4.711448] io scheduler bfq registered [ 4.712984] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.715269] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.717247] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.719858] ACPI: Power Button [PWRF] [ 4.724192] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.729362] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.739175] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.746636] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.768897] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.803719] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.841324] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.846792] Non-volatile memory driver v1.3 [ 4.848629] Linux agpgart interface v0.103 [ 4.886738] virtio_blk virtio1: [vda] 134160 512-byte logical blocks (68.7 MB/65.5 MiB) [ 4.893223] vda: detected capacity change from 0 to 68689920 [ 4.911225] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.915559] vdb: detected capacity change from 0 to 1073741824 [ 5.023099] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.027263] vdc: detected capacity change from 0 to 2621440000 [ 5.050621] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.053886] vdd: detected capacity change from 0 to 2621440000 [ 5.078447] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.082375] vde: detected capacity change from 0 to 4294967296 [ 5.193637] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.197196] vdf: detected capacity change from 0 to 4294967296 [ 5.209889] libphy: Fixed MDIO Bus: probed [ 5.220688] usbcore: registered new interface driver usbserial_generic [ 5.223505] usbserial: USB Serial support registered for generic [ 5.225559] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.230025] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.232727] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.236135] mousedev: PS/2 mouse device common for all mice [ 5.239942] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.244623] rtc_cmos 00:05: RTC can wake from S4 [ 5.248169] rtc_cmos 00:05: registered as rtc0 [ 5.250872] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.254520] intel_pstate: CPU model not supported [ 5.257779] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.262736] hid: raw HID events driver (C) Jiri Kosina [ 5.265203] usbcore: registered new interface driver usbhid [ 5.265485] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.269862] usbhid: USB HID core driver [ 5.270050] drop_monitor: Initializing network drop monitor service [ 5.270181] Initializing XFRM netlink socket [ 5.287656] NET: Registered protocol family 10 [ 5.376066] Segment Routing with IPv6 [ 5.377811] NET: Registered protocol family 17 [ 5.380731] mpls_gso: MPLS GSO support [ 5.386662] RAS: Correctable Errors collector initialized. [ 5.388464] AVX version of gcm_enc/dec engaged. [ 5.390447] AES CTR mode by8 optimization enabled [ 5.629862] sched_clock: Marking stable (5629775729, 0)->(6713234451, -1083458722) [ 5.633944] registered taskstats version 1 [ 5.635510] Loading compiled-in X.509 certificates [ 5.637618] zswap: loaded using pool lzo/zbud [ 5.667936] Key type big_key registered [ 5.708126] Key type encrypted registered [ 5.710148] ima: No TPM chip found, activating TPM-bypass! [ 5.713359] ima: Allocated hash algorithm: sha1 [ 5.716290] ima: No architecture policies found [ 5.717703] evm: Initialising EVM extended attributes: [ 5.720179] evm: security.selinux [ 5.721845] evm: security.ima [ 5.722857] evm: security.capability [ 5.724632] evm: HMAC attrs: 0x1 [ 5.730401] rtc_cmos 00:05: setting system clock to 2026-04-11 05:07:21 UTC (1775884041) [ 5.743407] debug: unmapping init [mem 0xffffffffa6603000-0xffffffffa67fffff] [ 5.823874] debug: unmapping init [mem 0xffffffffa5382000-0xffffffffa5658fff] [ 5.834348] Write protecting the kernel read-only data: 28672k [ 5.851896] debug: unmapping init [mem 0xffffffffa3a03000-0xffffffffa3bfffff] [ 5.861284] debug: unmapping init [mem 0xffffffffa4314000-0xffffffffa43fffff] [ 5.949261] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.963186] systemd[1]: Detected virtualization kvm. [ 5.965301] systemd[1]: Detected architecture x86-64. [ 5.967930] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.015699] systemd[1]: No hostname configured. [ 6.017827] systemd[1]: Set hostname to . [ 6.020209] random: systemd: uninitialized urandom read (16 bytes read) [ 6.023188] systemd[1]: Initializing machine ID from random generator. [ 6.242573] random: ln: uninitialized urandom read (6 bytes read) [ 6.416257] random: systemd: uninitialized urandom read (16 bytes read) [ 6.421407] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.437282] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 6.452902] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.047766] device-mapper: uevent: version 1.0.3 [ 9.049829] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Mounted Kernel Configuration File System. [ 10.876916] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target System Initialization. [ 11.127481] random: fast init done [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.390345] scsi host0: ata_piix [ 12.416978] scsi host1: ata_piix [ 12.418793] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.421780] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.236793] random: crng init done [ 19.238190] random: 7 urandom warning(s) missed due to ratelimiting [ 20.559724] dracut-initqueue[569]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 22.478092] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.747261] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.536959] SELinux: Disabled at runtime. [ 25.603647] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.613528] systemd[1]: Detected virtualization kvm. [ 25.615231] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.777132] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.785156] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.817670] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.833629] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.845441] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.869683] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.888556] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 26.961259] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 28.496514] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.253626] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.333469] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.061356] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.174654] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 35.723480] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 36.567754] NFS: Registering the id_resolver key type [ 36.570049] Key type id_resolver registered [ 36.571207] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ 42.042005] hrtimer: interrupt took 12990045 ns Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Permit User Sessions. [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg244-server login: [ 86.332556] libcfs: loading out-of-tree module taints kernel. [ 86.356443] Key type ._llcrypt registered [ 86.358563] Key type .llcrypt registered [ 86.413296] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 97.608223] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 98.481497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.493922] alg: No test for adler32 (adler32-zlib) [ 99.680683] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 100.141569] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 101.784160] Key type lgssc registered [ 103.077926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.859914] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 138.622330] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 146.330113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 146.357930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.582813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.609854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.706677] Lustre: lustre-MDT0000: new disk, initializing [ 147.775139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.792627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 150.500561] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.464348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.539499] Lustre: 6466:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 159.560126] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 159.563695] Lustre: Skipped 1 previous similar message [ 159.622893] Lustre: lustre-MDT0001: new disk, initializing [ 159.678818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.697618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.711482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 162.124484] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.499912] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 171.299097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 171.453284] Lustre: lustre-OST0000: new disk, initializing [ 171.456713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 171.495094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 175.224315] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.648607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 176.657690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 176.692241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 183.337290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 183.433165] Lustre: lustre-OST0001: new disk, initializing [ 183.439766] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 183.485467] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.757521] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.018451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 192.027298] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 192.059828] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 197.278964] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 207.754291] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 217.554610] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing check_logdir /tmp/testlogs/ [ 221.012750] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing yml_node [ 223.663700] Lustre: DEBUG MARKER: Client: 2.17.52.3 [ 225.521914] Lustre: DEBUG MARKER: MDS: 2.17.52.3 [ 227.166261] Lustre: DEBUG MARKER: OSS: 2.17.52.3 [ 228.256697] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Apr 11 01:11:03 EDT 2026 [ 239.012976] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 239.962875] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 253.414545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 253.415583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 253.420700] Lustre: Skipped 2 previous similar messages [ 253.423357] Lustre: Skipped 1 previous similar message [ 258.530078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 258.533234] Lustre: Skipped 3 previous similar messages [ 263.651132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 263.655251] Lustre: Skipped 3 previous similar messages [ 267.232143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 267.355440] Lustre: server umount lustre-MDT0000 complete [ 269.751127] LustreError: 6461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884305 with bad export cookie 2886279444798088933 [ 269.757832] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 269.760853] LustreError: 6461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 273.888948] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.891065] LustreError: 9834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 273.894777] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 273.903818] Lustre: Skipped 2 previous similar messages [ 273.916299] LustreError: 9834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 274.914660] LustreError: 9834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 274.924830] LustreError: 9834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 280.033926] LustreError: 10253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.033926] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 280.055101] Lustre: Skipped 4 previous similar messages [ 284.132431] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 284.269686] Lustre: server umount lustre-MDT0001 complete [ 286.820996] Lustre: server umount lustre-OST0000 complete [ 289.799487] Lustre: server umount lustre-OST0001 complete [ 294.053370] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 300.424975] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 320.971205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 321.108922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 321.122376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 321.163788] Lustre: lustre-MDT0000: new disk, initializing [ 321.213421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 321.226877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 323.610666] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 330.321625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 330.392767] Lustre: 14935:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 330.432701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 330.434823] Lustre: Skipped 1 previous similar message [ 330.494497] Lustre: lustre-MDT0001: new disk, initializing [ 330.586320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 330.625944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 330.636184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 333.190716] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 338.156553] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 340.402986] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 344.472093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 346.331413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 346.407510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 348.055939] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 352.654529] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 355.052951] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 355.206114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 357.753952] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 357.892503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 361.958178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.959155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 361.960296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 361.965712] Lustre: Skipped 1 previous similar message [ 365.228599] Lustre: server umount lustre-OST0000 complete [ 371.682374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 371.685979] LustreError: Skipped 1 previous similar message [ 371.688679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.445529] Lustre: server umount lustre-MDT0000 complete [ 375.761147] LustreError: 15721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884411 with bad export cookie 2886279444798090473 [ 375.769958] LustreError: 15721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 375.770044] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.991493] Lustre: server umount lustre-MDT0001 complete [ 378.551869] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:13:33 (1775884413) [ 380.044818] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 381.025886] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:13:35 (1775884415) [ 385.216159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.466790] LustreError: 18940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 385.478740] LustreError: 18940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 385.530542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 385.535578] Lustre: Skipped 1 previous similar message [ 387.845297] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 390.627414] LustreError: 18941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 392.726167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.885527] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 398.556743] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 400.467320] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 404.471180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 408.314259] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 411.799624] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 426.474073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 426.478326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 426.487739] Lustre: Skipped 1 previous similar message [ 426.497298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 426.501182] Lustre: Skipped 4 previous similar messages [ 431.084915] Lustre: server umount lustre-OST0000 complete [ 439.425121] Lustre: server umount lustre-MDT0000 complete [ 441.837030] LustreError: 18922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884477 with bad export cookie 2886279444798091355 [ 441.838449] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 441.861835] LustreError: 18922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 442.040141] Lustre: server umount lustre-MDT0001 complete [ 447.701430] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 449.428842] Key type lgssc unregistered [ 449.601743] LNet: 22138:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.608947] LNetError: 22138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 449.626374] LNet: Removed LNI 192.168.202.144@tcp [ 450.168254] Key type .llcrypt unregistered [ 450.173810] Key type ._llcrypt unregistered [ 463.989462] Key type ._llcrypt registered [ 463.991518] Key type .llcrypt registered [ 464.067562] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 465.012381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 465.037461] alg: No test for adler32 (adler32-zlib) [ 466.031830] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 466.156734] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 467.768430] Key type lgssc registered [ 468.494517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 473.343250] loop: module loaded [ 473.395211] loop0: detected capacity change from 0 to 204800000 [ 473.559945] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 473.573601] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 473.780757] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 477.370294] loop0: detected capacity change from 0 to 204800000 [ 477.543120] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 477.549450] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 477.561280] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 480.950243] loop0: detected capacity change from 0 to 204800000 [ 484.695299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 484.707456] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.880670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 485.918760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 485.973575] Lustre: lustre-MDT0000: new disk, initializing [ 486.031871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 486.045302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 488.880908] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 495.384663] loop1: detected capacity change from 0 to 204800000 [ 499.282650] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 499.495785] Lustre: lustre-OST07c6: new disk, initializing [ 499.499800] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 499.507277] Lustre: Skipped 1 previous similar message [ 499.567122] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 503.423346] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 507.439963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 507.448035] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 507.475164] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 513.049145] Lustre: 24267:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 513.054668] Lustre: 24267:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 513.058944] Lustre: 24267:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 513.063455] Lustre: 24267:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 513.068152] Lustre: 24267:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 513.073076] Lustre: 24267:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 533.220206] Lustre: 24266:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 533.224041] Lustre: 24266:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 533.227038] Lustre: 24266:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 533.231862] Lustre: 24266:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 533.235800] Lustre: 24266:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 533.239387] Lustre: 24266:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 537.354673] Lustre: 24268:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 537.361297] Lustre: 24268:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 537.366032] Lustre: 24268:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 537.370757] Lustre: 24268:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.374950] Lustre: 24268:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 537.379388] Lustre: 24268:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.382847] Lustre: 24268:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 537.386690] Lustre: 24268:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.389154] Lustre: 24268:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 537.395054] Lustre: 24268:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 537.398376] Lustre: 24268:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 537.402314] Lustre: 24268:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 540.487757] Lustre: 24266:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 540.500881] Lustre: 24266:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 540.505482] Lustre: 24266:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 540.509991] Lustre: 24266:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 540.515172] Lustre: 24266:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 540.519437] Lustre: 24266:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 540.524341] Lustre: 24266:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 540.529404] Lustre: 24266:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 540.533723] Lustre: 24266:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 540.538438] Lustre: 24266:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 540.543145] Lustre: 24266:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 540.547722] Lustre: 24266:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 544.492674] Lustre: 24267:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 544.505608] Lustre: 24267:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 544.509739] Lustre: 24267:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 544.517222] Lustre: 24267:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 544.523346] Lustre: 24267:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 544.531379] Lustre: 24267:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 544.537718] Lustre: 24267:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 544.545466] Lustre: 24267:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 544.550291] Lustre: 24267:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 544.556312] Lustre: 24267:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 544.563030] Lustre: 24267:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 544.568664] Lustre: 24267:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 555.002724] Lustre: 24268:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 555.008158] Lustre: 24268:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 5 previous similar messages [ 555.010870] Lustre: 24268:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 555.019838] Lustre: 24268:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 555.024638] Lustre: 24268:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 555.030628] Lustre: 24268:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 555.037148] Lustre: 24268:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 555.043171] Lustre: 24268:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 555.048317] Lustre: 24268:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 555.053655] Lustre: 24268:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 555.059188] Lustre: 24268:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 555.064639] Lustre: 24268:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 602.893645] Lustre: server umount lustre-OST07c6 complete [ 617.557604] Lustre: server umount lustre-MDT0000 complete [ 626.931038] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 628.597822] Key type lgssc unregistered [ 628.809101] LNet: 27787:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.814776] LNetError: 27787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.863632] LNet: Removed LNI 192.168.202.144@tcp [ 630.379191] Key type .llcrypt unregistered [ 630.384519] Key type ._llcrypt unregistered [ 641.318545] Key type ._llcrypt registered [ 641.320519] Key type .llcrypt registered [ 641.379623] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 641.974128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 642.051229] alg: No test for adler32 (adler32-zlib) [ 642.980544] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 643.106582] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 644.712216] Key type lgssc registered [ 645.365483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 649.885368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 649.904619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.237921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 653.531187] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 658.671371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.880910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 661.400581] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 664.777746] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 666.831740] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 671.196537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 671.404857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 675.535632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 675.679753] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 679.238126] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 693.732259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 693.732569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.740173] LustreError: Skipped 1 previous similar message [ 693.751239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 694.242484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 694.249750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.261076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 694.264897] Lustre: Skipped 1 previous similar message [ 699.362519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 699.365868] Lustre: Skipped 1 previous similar message [ 699.453243] Lustre: server umount lustre-OST0000 complete [ 704.484210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 704.494976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 704.498312] Lustre: Skipped 1 previous similar message [ 709.602948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.611209] Lustre: Skipped 1 previous similar message [ 716.257017] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 716.377084] Lustre: server umount lustre-MDT0000 complete [ 718.346201] LustreError: 29001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884754 with bad export cookie 16023306832303266419 [ 718.349980] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 718.352056] LustreError: 29001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 718.539888] Lustre: server umount lustre-MDT0001 complete [ 724.649180] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 726.406508] Key type lgssc unregistered [ 726.620366] LNet: 32399:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.625888] LNetError: 32399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.655660] LNet: Removed LNI 192.168.202.144@tcp [ 728.218813] Key type .llcrypt unregistered [ 728.221066] Key type ._llcrypt unregistered [ 739.290341] Key type ._llcrypt registered [ 739.292259] Key type .llcrypt registered [ 739.356067] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 740.024761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 740.036454] alg: No test for adler32 (adler32-zlib) [ 740.978646] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 741.114114] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 742.728273] Key type lgssc registered [ 743.464260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 747.945646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 747.960274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 749.301063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.495394] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 756.041455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 756.238550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 758.444765] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 761.889381] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 763.942440] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 767.699125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 767.906387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 771.099730] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 774.340727] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 775.089583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 785.377207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 785.380553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 785.389109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 786.404425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 786.407092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 786.416482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.021556] Lustre: server umount lustre-OST0000 complete [ 796.641238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.645499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 796.649033] Lustre: Skipped 1 previous similar message [ 796.655186] Lustre: Skipped 2 previous similar messages [ 801.764177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 801.769758] Lustre: Skipped 1 previous similar message [ 806.885243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 806.895929] Lustre: Skipped 1 previous similar message [ 807.392304] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 807.520265] Lustre: server umount lustre-MDT0000 complete [ 809.749366] LustreError: 33610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884845 with bad export cookie 2560750359016419019 [ 809.750905] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 809.754622] LustreError: 33610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 809.942498] Lustre: server umount lustre-MDT0001 complete [ 816.088042] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 817.702248] Key type lgssc unregistered [ 817.905904] LNet: 36810:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.910695] LNetError: 36810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.922440] LNet: Removed LNI 192.168.202.144@tcp [ 818.380253] Key type .llcrypt unregistered [ 818.382192] Key type ._llcrypt unregistered [ 831.412409] Key type ._llcrypt registered [ 831.416381] Key type .llcrypt registered [ 831.482524] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 831.938114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 832.019826] alg: No test for adler32 (adler32-zlib) [ 832.987403] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 833.116394] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 834.712270] Key type lgssc registered [ 835.476799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 840.206942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 840.220257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 841.501215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 843.700370] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 848.583604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.845895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 851.552574] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 854.973962] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 857.274835] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 861.338582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 861.539197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 865.006784] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 866.670112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 868.246290] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 879.079302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.082544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 879.090956] Lustre: Skipped 2 previous similar messages [ 879.098759] Lustre: Skipped 1 previous similar message [ 880.012262] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.44@tcp (stopping) [ 880.015284] Lustre: Skipped 1 previous similar message [ 882.597941] Lustre: server umount lustre-MDT0000 complete [ 884.198245] LustreError: 38182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 884.210828] LustreError: 38182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 885.122196] LustreError: 40340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 885.144634] LustreError: 38167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884920 with bad export cookie 3137783707182654925 [ 885.148036] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 885.152903] LustreError: 38167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 889.317020] LustreError: 38182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 889.321332] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.335585] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 891.438978] Lustre: server umount lustre-MDT0001 complete [ 910.657427] Lustre: server umount lustre-OST0000 complete [ 918.949968] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 920.578467] Key type lgssc unregistered [ 920.784591] LNet: 41692:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.788412] LNetError: 41692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.796668] LNet: Removed LNI 192.168.202.144@tcp [ 921.217484] Key type .llcrypt unregistered [ 921.219773] Key type ._llcrypt unregistered [ 933.757669] Key type ._llcrypt registered [ 933.759066] Key type .llcrypt registered [ 933.812603] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 934.272956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 934.353115] alg: No test for adler32 (adler32-zlib) [ 935.275047] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 935.422071] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 937.040221] Key type lgssc registered [ 937.606526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 941.914443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 941.932518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.184036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 945.499380] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 950.275344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 950.499257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 952.628547] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 955.370791] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 957.194470] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 960.890858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 961.118665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.064694] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 966.933505] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 967.275156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 983.009250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 983.013340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.020351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.963635] Lustre: lustre-OST0000: Not available for connect from 192.168.202.44@tcp (stopping) [ 986.084240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 986.087770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 986.089680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.098050] Lustre: Skipped 1 previous similar message [ 987.301662] Lustre: server umount lustre-OST0000 complete [ 1001.442787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.444065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1001.448066] Lustre: Skipped 1 previous similar message [ 1005.537850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1005.543136] Lustre: Skipped 2 previous similar messages [ 1007.583410] Lustre: server umount lustre-MDT0000 complete [ 1009.480423] LustreError: 43050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885045 with bad export cookie 15813663093001467983 [ 1009.480987] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1009.488058] LustreError: 43050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1015.722405] Lustre: server umount lustre-MDT0001 complete [ 1020.324401] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 1021.616158] Key type lgssc unregistered [ 1021.772381] LNet: 46446:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.776262] LNetError: 46446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.789427] LNet: Removed LNI 192.168.202.144@tcp [ 1022.193718] Key type .llcrypt unregistered [ 1022.195192] Key type ._llcrypt unregistered [ 1033.790838] Key type ._llcrypt registered [ 1033.792519] Key type .llcrypt registered [ 1033.841238] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1034.349910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1034.360883] alg: No test for adler32 (adler32-zlib) [ 1035.247987] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1035.361730] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 1036.960209] Key type lgssc registered [ 1037.448065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1040.967606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1040.976642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.151516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1043.693790] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1047.604329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.790818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1049.626318] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1052.318738] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1054.157948] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1057.741189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1057.935528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1060.666674] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1062.965843] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1067.178636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1072.130748] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1073.922587] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1084.889106] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-11 1h25m19s [ 1085.556877] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1087.971665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.972878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1087.976181] Lustre: Skipped 1 previous similar message [ 1087.990204] Lustre: Skipped 2 previous similar messages [ 1089.063445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1090.939169] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.44@tcp (stopping) [ 1090.944217] Lustre: Skipped 3 previous similar messages [ 1092.599986] Lustre: server umount lustre-MDT0000 complete [ 1094.276618] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1095.138826] LustreError: 47823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.147354] LustreError: 47823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1095.990939] LustreError: 47818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1097.279787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.340595] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1097.458211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1097.492050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1098.471173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1098.973596] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1100.320093] Lustre: DEBUG MARKER: Wait for df (36030) ... [ 1101.035986] Lustre: DEBUG MARKER: done [ 1102.818685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1102.822986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.829849] Lustre: Skipped 1 previous similar message [ 1102.836061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1103.845771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.556470] Lustre: server umount lustre-OST0000 complete [ 1108.964833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1108.964870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.966721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1108.966730] Lustre: Skipped 2 previous similar messages [ 1108.976540] Lustre: Skipped 1 previous similar message [ 1112.735926] Lustre: server umount lustre-MDT0000 complete [ 1114.324762] LustreError: 47804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885150 with bad export cookie 572003376630009598 [ 1114.325212] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1114.329324] LustreError: 47804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1114.468054] Lustre: server umount lustre-MDT0001 complete [ 1118.611711] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 1119.725795] Key type lgssc unregistered [ 1119.863322] LNet: 52367:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.867528] LNetError: 52367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1119.875320] LNet: Removed LNI 192.168.202.144@tcp [ 1120.198750] Key type .llcrypt unregistered [ 1120.200553] Key type ._llcrypt unregistered [ 1135.963070] Key type ._llcrypt registered [ 1135.964751] Key type .llcrypt registered [ 1136.007748] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1136.453942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1136.478161] alg: No test for adler32 (adler32-zlib) [ 1137.339267] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1137.433994] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 1139.024160] Key type lgssc registered [ 1139.471102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1142.459583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1142.469928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.626846] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1143.636935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1143.708104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.273781] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1148.747099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.802176] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1148.815105] Lustre: 54535:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1148.926522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1150.613347] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1152.856438] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1154.157658] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1156.716841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1156.829887] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1156.921118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.296193] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1159.449868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1161.446797] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1166.321825] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-11 1h26m41s [ 1167.038457] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1170.216228] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1171.267262] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1171.285375] LustreError: 54541:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1191.305673] Lustre: lustre-MDT0000: Client 18cc5421-cba8-4372-9c0f-d28ef01bf35c (at 192.168.202.44@tcp) reconnecting [ 1211.328097] LustreError: 54541:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1211.347529] Lustre: lustre-MDT0000: Client 18cc5421-cba8-4372-9c0f-d28ef01bf35c (at 192.168.202.44@tcp) reconnecting [ 1212.188278] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1213.245384] Lustre: DEBUG MARKER: done [ 1215.970977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1215.971088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.971677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1215.974139] LustreError: Skipped 1 previous similar message [ 1215.978799] Lustre: Skipped 1 previous similar message [ 1221.090193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1221.092648] Lustre: Skipped 2 previous similar messages [ 1221.418455] Lustre: server umount lustre-OST0000 complete [ 1225.697100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1225.700608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.707942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1229.154698] Lustre: server umount lustre-MDT0000 complete [ 1230.616082] LustreError: 54526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885266 with bad export cookie 16422118777774052504 [ 1230.616952] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.620530] LustreError: 54526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1230.822300] Lustre: server umount lustre-MDT0001 complete [ 1234.953152] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 1236.171308] Key type lgssc unregistered [ 1236.317486] LNet: 58498:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.320596] LNetError: 58498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.329516] LNet: Removed LNI 192.168.202.144@tcp [ 1236.699824] Key type .llcrypt unregistered [ 1236.701611] Key type ._llcrypt unregistered [ 1256.510111] Key type ._llcrypt registered [ 1256.511614] Key type .llcrypt registered [ 1256.557636] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1256.963426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1256.989833] alg: No test for adler32 (adler32-zlib) [ 1257.894997] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1258.009156] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 1259.592165] Key type lgssc registered [ 1260.051449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1261.482886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1261.494247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.617106] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1262.626985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1262.714831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.109503] Lustre: Failing over lustre-MDT0000 [ 1263.241716] Lustre: server umount lustre-MDT0000 complete [ 1265.894591] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:28:21 (1775885301) [ 1268.514054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.674930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1270.074774] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1273.572655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.638646] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1273.663180] Lustre: 61258:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1273.722700] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1273.814712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1275.300159] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1277.540062] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1279.063234] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1281.970976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1282.064597] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1282.148979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.478603] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1286.680863] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1288.223925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1295.376727] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1298.402401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.404258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1298.408503] Lustre: Skipped 2 previous similar messages [ 1298.411202] Lustre: Skipped 1 previous similar message [ 1303.522152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1303.524975] Lustre: Skipped 3 previous similar messages [ 1303.914982] Lustre: server umount lustre-MDT0000 complete [ 1305.632699] LustreError: 61250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885341 with bad export cookie 12602952239151157461 [ 1305.635546] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1305.638410] LustreError: 61250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1305.769091] Lustre: server umount lustre-MDT0001 complete [ 1307.201293] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1309.067972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1312.700483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.808689] LustreError: 60179:0:(client.c:1393:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000230f3068 x1862150697026560/t0(0) o250->MGC192.168.202.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1315.963868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1315.990602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1317.365032] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1320.685068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.789113] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.284474] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1324.356473] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1324.896211] Lustre: 60180:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775885344/real 1775885344] req@00000000bec0ee9f x1862150697026176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775885360 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1325.752436] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1326.056122] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1327.056170] Lustre: 60180:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775885346/real 1775885346] req@000000006960d8f9 x1862150697026304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775885362 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1329.126729] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1330.144923] Lustre: 60183:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775885349/real 1775885349] req@000000004943d539 x1862150697026432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775885365 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1332.192439] Lustre: 60182:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775885351/real 1775885351] req@0000000041719603 x1862150697026688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775885367 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1334.242969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.244184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.248320] Lustre: Skipped 1 previous similar message [ 1334.250864] Lustre: Skipped 1 previous similar message [ 1339.227273] Lustre: server umount lustre-MDT0000 complete [ 1339.361489] LustreError: 64683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.369318] LustreError: 64683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1340.885105] LustreError: 61251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885376 with bad export cookie 12602952239151159743 [ 1340.885631] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.889246] LustreError: 61251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1341.025701] Lustre: server umount lustre-MDT0001 complete [ 1343.837136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1348.038325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.624763] LustreError: 60179:0:(client.c:1393:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003a595532 x1862150697047424/t0(0) o250->MGC192.168.202.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1350.746697] LustreError: 64704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.781710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.783795] Lustre: Skipped 1 previous similar message [ 1350.804522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1352.480352] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1356.104388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1356.222267] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.721437] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1359.906776] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1360.352823] Lustre: 60182:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775885380/real 1775885380] req@00000000b626795a x1862150697047040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775885396 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1361.383596] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1361.413525] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1364.514845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.515103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1364.519811] Lustre: Skipped 1 previous similar message [ 1364.522065] Lustre: Skipped 2 previous similar messages [ 1369.645393] Lustre: server umount lustre-MDT0000 complete [ 1371.083302] LustreError: 61249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885406 with bad export cookie 12602952239151161122 [ 1371.086280] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1371.087550] LustreError: 61249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1371.222784] Lustre: server umount lustre-MDT0001 complete [ 1373.702538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1374.935539] Lustre: DEBUG MARKER: files compared the same [ 1386.028960] Lustre: server umount lustre-OST0000 complete [ 1392.481225] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 1393.878928] Key type lgssc unregistered [ 1394.047613] LNet: 70169:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.052380] LNetError: 70169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.065958] LNet: Removed LNI 192.168.202.144@tcp [ 1394.479399] Key type .llcrypt unregistered [ 1394.481157] Key type ._llcrypt unregistered [ 1405.924752] Key type ._llcrypt registered [ 1405.926314] Key type .llcrypt registered [ 1405.979717] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1406.450648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1406.463333] alg: No test for adler32 (adler32-zlib) [ 1407.366161] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1407.471904] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 1409.088726] Key type lgssc registered [ 1409.711205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1413.786840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1413.799834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.054344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1416.964957] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug +malloc all [ 1420.900583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.068033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1423.056331] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug +malloc all [ 1425.559778] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1427.284336] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1430.152725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1430.309568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1432.963702] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug +malloc all [ 1435.547576] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1439.723189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1444.833534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.833709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1444.833844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1444.839134] Lustre: Skipped 1 previous similar message [ 1449.952783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.960468] Lustre: Skipped 2 previous similar messages [ 1455.074075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1455.076734] Lustre: Skipped 1 previous similar message [ 1459.168179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1459.230711] Lustre: server umount lustre-OST0000 complete [ 1462.240992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1462.249606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.261307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1465.313485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.198310] Lustre: server umount lustre-MDT0000 complete [ 1468.774145] LustreError: 73699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885504 with bad export cookie 17558213947907057272 [ 1468.774557] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1468.778560] LustreError: 73699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1468.901587] Lustre: server umount lustre-MDT0001 complete [ 1473.089662] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 1474.414489] Key type lgssc unregistered [ 1474.569736] LNet: 74723:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.573634] LNetError: 74723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1474.582838] LNet: Removed LNI 192.168.202.144@tcp [ 1474.956752] Key type .llcrypt unregistered [ 1474.958385] Key type ._llcrypt unregistered [ 1485.767355] Key type ._llcrypt registered [ 1485.768638] Key type .llcrypt registered [ 1485.826624] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1486.210937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1486.249032] alg: No test for adler32 (adler32-zlib) [ 1487.149384] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1487.270056] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 1488.880146] Key type lgssc registered [ 1489.320519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1492.289079] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1492.296769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1507.744425] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1512.864322] LustreError: 76075:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.144@tcp: failed processing log, type 4: rc = -110 [ 1539.488344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1541.612695] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1543.762883] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1547.430344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1547.495391] Lustre: *** cfs_fail_loc=706, val=0*** [ 1547.684428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.734229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1549.376584] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1552.873906] LustreError: 76101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1552.880392] LustreError: 76101:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1553.205358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1553.365444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1555.054985] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1557.321766] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1558.803016] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1563.104943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1563.108466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1563.113934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1563.617278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1563.619953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1563.625607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1564.643241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1564.648829] Lustre: Skipped 1 previous similar message [ 1569.763189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.766024] Lustre: Skipped 1 previous similar message [ 1574.368141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1574.423417] Lustre: server umount lustre-OST0000 complete [ 1578.976982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1578.980155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1578.985144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1581.851989] Lustre: server umount lustre-MDT0000 complete [ 1583.012402] LustreError: 77052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1583.021843] LustreError: 77052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1583.360281] LustreError: 76082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885619 with bad export cookie 6990293133436904467 [ 1583.362372] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1583.365320] LustreError: 76082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1583.495978] Lustre: server umount lustre-MDT0001 complete [ 1587.861505] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 1589.172731] Key type lgssc unregistered [ 1589.313318] LNet: 79336:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.315889] LNetError: 79336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.323544] LNet: Removed LNI 192.168.202.144@tcp [ 1589.653071] Key type .llcrypt unregistered [ 1589.654272] Key type ._llcrypt unregistered [ 1599.697125] Key type ._llcrypt registered [ 1599.698520] Key type .llcrypt registered [ 1599.736502] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1600.199054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1600.205992] alg: No test for adler32 (adler32-zlib) [ 1601.095157] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1601.186834] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 1602.776178] Key type lgssc registered [ 1603.182087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1606.195232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1606.207080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1608.804677] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1611.858622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1612.106121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1613.941770] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1617.377989] LustreError: 81383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1617.389134] LustreError: 81383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1618.120528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1618.291227] LustreError: 81382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1618.302831] LustreError: 81382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1618.303209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.766742] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1621.411694] LustreError: 81387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.422492] LustreError: 81387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1624.253446] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1624.258324] Lustre: MGS: Client 68772e2d-3133-4d92-9ca6-0297dcdeaeb8 (at 0@lo) reconnecting [ 1624.358237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.387612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1625.785431] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1634.789814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1634.790734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1634.791327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1634.795477] Lustre: Skipped 1 previous similar message [ 1639.909430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1639.913324] Lustre: Skipped 1 previous similar message [ 1640.570789] Lustre: server umount lustre-OST0000 complete [ 1645.025471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1645.026707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.030294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1645.036693] Lustre: Skipped 1 previous similar message [ 1645.040165] Lustre: Skipped 2 previous similar messages [ 1648.330874] Lustre: server umount lustre-MDT0000 complete [ 1650.141663] Lustre: server umount lustre-MDT0001 complete [ 1651.828279] Lustre: server umount MGS complete [ 1656.625476] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:34:51 (1775885691) [ 1668.742030] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 1672.978522] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1687.866692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1689.700053] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1694.347316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1694.439622] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1694.462322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1694.499812] Lustre: lustre-MDT0001: new disk, initializing [ 1694.529552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1694.540067] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1694.544494] Lustre: 88445:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1694.548595] LustreError: 88445:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1696.039986] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1700.700191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1700.793562] Lustre: lustre-OST0000: new disk, initializing [ 1700.796122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1700.822137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.865546] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1706.211467] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1708.641025] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1708.648360] Lustre: MGS: Client 038827d6-6925-4985-81cd-5cf296ab326d (at 0@lo) reconnecting [ 1708.665191] Lustre: 90126:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1708.674311] LustreError: 88430:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1708.678176] LustreError: 88430:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1708.692514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1708.734030] Lustre: lustre-MDT0000: new disk, initializing [ 1708.810546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1709.090247] Lustre: 90037:0:(client.c:1640:after_reply()) @@@ resending request on EINPROGRESS req@00000000efc72a64 x1862151056694016/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1775885760 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1709.856874] Lustre: 90152:0:(client.c:1640:after_reply()) @@@ resending request on EINPROGRESS req@00000000c273e44d x1862151056694144/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1775885761 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1710.550756] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1711.136773] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1718.181355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1718.185214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1718.948789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1719.017441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1741.282048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.288722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1741.289896] Lustre: Skipped 1 previous similar message [ 1741.295969] Lustre: Skipped 1 previous similar message [ 1746.264072] Lustre: server umount lustre-OST0000 complete [ 1751.521912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1751.522152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.527851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1751.531326] Lustre: Skipped 1 previous similar message [ 1753.939802] Lustre: server umount lustre-MDT0000 complete [ 1761.789830] Lustre: server umount lustre-MDT0001 complete [ 1763.229990] Lustre: server umount MGS complete [ 1767.372591] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:36:42 (1775885802) [ 1774.215955] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 1775.416057] Key type lgssc unregistered [ 1775.565971] LNet: 92614:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.569764] LNetError: 92614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.578414] LNet: Removed LNI 192.168.202.144@tcp [ 1775.916342] Key type .llcrypt unregistered [ 1775.917981] Key type ._llcrypt unregistered [ 1784.013710] Key type ._llcrypt registered [ 1784.015102] Key type .llcrypt registered [ 1784.062436] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1784.544415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1784.551771] alg: No test for adler32 (adler32-zlib) [ 1785.410851] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1785.498430] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 1787.080182] Key type lgssc registered [ 1787.464966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1790.304113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1790.310510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.483430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.961193] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1795.928354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.090925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1797.549322] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1799.350645] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1800.530879] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1803.003433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1803.178947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1805.331575] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1807.315297] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1816.494714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1821.667163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.668793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1821.670101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1821.676503] Lustre: Skipped 1 previous similar message [ 1822.748959] Lustre: server umount lustre-OST0000 complete [ 1826.456300] Lustre: server umount lustre-MDT0000 complete [ 1826.787156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1826.788114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1826.788923] LustreError: 93796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.788929] LustreError: 93796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1826.793228] Lustre: Skipped 1 previous similar message [ 1826.796095] LustreError: Skipped 1 previous similar message [ 1827.804048] LustreError: 93775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885863 with bad export cookie 12230191713376118967 [ 1827.807789] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.808694] LustreError: 93775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1827.925172] Lustre: server umount lustre-MDT0001 complete [ 1831.897353] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 1832.946912] Key type lgssc unregistered [ 1833.078372] LNet: 96992:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.082190] LNetError: 96992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.092335] LNet: Removed LNI 192.168.202.144@tcp [ 1833.395555] Key type .llcrypt unregistered [ 1833.397068] Key type ._llcrypt unregistered [ 1840.077647] Key type ._llcrypt registered [ 1840.078660] Key type .llcrypt registered [ 1840.128688] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1840.477083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1840.500075] alg: No test for adler32 (adler32-zlib) [ 1841.348542] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1841.435811] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 1843.024170] Key type lgssc registered [ 1843.404737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1845.746213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1845.749598] LustreError: 98034:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 1845.754107] LustreError: 98037:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 1845.756596] LustreError: 98034:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1845.759459] LustreError: 98034:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1845.762856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.764491] LustreError: 98034:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1846.930411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.541752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.633989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1850.931706] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1852.550653] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing lsmod [ 1853.540344] LustreError: 99207:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 1853.545612] LustreError: 99208:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 1853.548921] LustreError: 99207:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1853.552668] LustreError: 99207:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1853.555792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1853.557380] LustreError: 99207:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1853.680708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.665111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.666976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1854.670940] Lustre: Skipped 1 previous similar message [ 1859.809943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1859.813420] Lustre: Skipped 4 previous similar messages [ 1860.830902] Lustre: server umount lustre-MDT0000 complete [ 1862.178141] LustreError: 98052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885897 with bad export cookie 15008074762197826670 [ 1862.181437] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1862.183315] LustreError: 98052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1863.904334] LustreError: 98505:0:(client.c:1383:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000048f3269b x1862151309376640/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1863.971266] Lustre: server umount lustre-MDT0001 complete [ 1875.688822] Lustre: server umount lustre-OST0000 complete [ 1878.512970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.668860] LustreError: 100277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1878.675995] LustreError: 100277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1878.692381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.965215] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1883.053642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.192823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1884.555337] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1886.427203] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1887.658856] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1889.968643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1891.876672] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1893.607559] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1904.423339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1913.825368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1913.829566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1913.835589] Lustre: Skipped 1 previous similar message [ 1913.838435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1914.850377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.851437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1916.692174] Lustre: server umount lustre-OST0000 complete [ 1918.945014] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1918.947988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.952269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1918.954062] Lustre: Skipped 2 previous similar messages [ 1924.313247] Lustre: server umount lustre-MDT0000 complete [ 1925.091127] LustreError: 100277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1925.097195] LustreError: 100277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1925.761471] LustreError: 100258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885961 with bad export cookie 15008074762197827342 [ 1925.765410] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1925.765856] LustreError: 100258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1925.898469] Lustre: server umount lustre-MDT0001 complete [ 1929.774065] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 1930.712711] Key type lgssc unregistered [ 1930.836347] LNet: 103474:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.838428] LNetError: 103474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1930.847287] LNet: Removed LNI 192.168.202.144@tcp [ 1931.124028] Key type .llcrypt unregistered [ 1931.125125] Key type ._llcrypt unregistered [ 1940.628307] Key type ._llcrypt registered [ 1940.629825] Key type .llcrypt registered [ 1940.668228] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1940.989647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1941.012071] alg: No test for adler32 (adler32-zlib) [ 1941.859164] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1941.937347] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 1943.512129] Key type lgssc registered [ 1943.872841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1946.561398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1946.569963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.755856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1949.222728] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1952.409537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.523199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1953.800144] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1955.671608] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1956.961109] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1959.562148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1959.700214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.739428] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 1963.749278] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1973.034659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1976.158227] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1976.976582] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1978.338122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1978.339045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.340717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1978.344564] Lustre: Skipped 1 previous similar message [ 1980.049908] Lustre: server umount lustre-OST0000 complete [ 1983.384760] Lustre: server umount lustre-MDT0000 complete [ 1983.458160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.462507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1983.462799] LustreError: 104844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.462804] LustreError: 104844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1983.462994] Lustre: Skipped 1 previous similar message [ 1983.465871] LustreError: Skipped 1 previous similar message [ 1984.652693] LustreError: 105569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886020 with bad export cookie 9383129616969523305 [ 1984.652693] LustreError: 104831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886020 with bad export cookie 9383129616969523305 [ 1984.654332] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1984.656605] LustreError: 105569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1984.774732] Lustre: server umount lustre-MDT0001 complete [ 1988.207257] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 1989.252471] Key type lgssc unregistered [ 1989.387490] LNet: 108154:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.391119] LNetError: 108154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1989.400405] LNet: Removed LNI 192.168.202.144@tcp [ 1989.681866] Key type .llcrypt unregistered [ 1989.683305] Key type ._llcrypt unregistered [ 1996.778497] Key type ._llcrypt registered [ 1996.780773] Key type .llcrypt registered [ 1996.843131] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 1997.241235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1997.283038] alg: No test for adler32 (adler32-zlib) [ 1998.146591] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1998.251266] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 1999.832177] Key type lgssc registered [ 2000.241692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2003.080154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2003.086230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.233485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2005.540422] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2008.562728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.681876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2010.048876] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2011.859714] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2013.192707] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2015.506934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2015.633657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.473253] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2019.166439] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2028.968081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2034.145759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2034.148305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.148840] LustreError: Skipped 1 previous similar message [ 2034.154965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2034.157433] Lustre: Skipped 1 previous similar message [ 2038.673988] Lustre: server umount lustre-OST0000 complete [ 2044.387940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.389977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2044.392541] Lustre: Skipped 2 previous similar messages [ 2044.396769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2046.167532] Lustre: server umount lustre-MDT0000 complete [ 2047.448071] LustreError: 109315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886083 with bad export cookie 13442688107077771507 [ 2047.451331] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.452213] LustreError: 109315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2047.572261] Lustre: server umount lustre-MDT0001 complete [ 2051.257478] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 2052.339337] Key type lgssc unregistered [ 2052.472462] LNet: 112534:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.478904] LNetError: 112534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2052.486340] LNet: Removed LNI 192.168.202.144@tcp [ 2052.767551] Key type .llcrypt unregistered [ 2052.769210] Key type ._llcrypt unregistered [ 2062.360743] Key type ._llcrypt registered [ 2062.361817] Key type .llcrypt registered [ 2062.404697] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2062.800685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.827517] alg: No test for adler32 (adler32-zlib) [ 2063.690603] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 2063.778203] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 2065.360135] Key type lgssc registered [ 2065.821443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2069.264612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2069.271530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.434055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.797073] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2075.131697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.279573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2076.850236] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2078.802651] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2080.077222] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2082.356733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2082.486339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.354363] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2086.221669] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2095.783488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2100.130947] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2101.505770] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2101.509199] Lustre: Skipped 2 previous similar messages [ 2103.155686] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2103.158067] Lustre: Skipped 2 previous similar messages [ 2109.439088] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2109.448497] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2112.173084] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.44@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2112.177482] Lustre: Skipped 2 previous similar messages [ 2115.840778] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.44@tcp 192.168.0.[2,10]@tcp [ 2117.754305] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.44/24@tcp in log lustre-MDT0000 [ 2117.758772] Lustre: Skipped 2 previous similar messages [ 2124.280558] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.44/24@tcp [ 2124.282746] Lustre: Skipped 1 previous similar message [ 2126.821845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2126.823242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2126.827354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2126.829183] LustreError: Skipped 1 previous similar message [ 2126.833132] Lustre: Skipped 1 previous similar message [ 2128.090977] Lustre: server umount lustre-OST0000 complete [ 2131.543932] Lustre: server umount lustre-MDT0000 complete [ 2131.937935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.938718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2131.940475] LustreError: 113911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.940481] LustreError: 113911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2131.941962] Lustre: Skipped 2 previous similar messages [ 2132.848413] LustreError: 113892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886168 with bad export cookie 14125792062534173574 [ 2132.851384] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2132.852659] LustreError: 113892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2132.971279] Lustre: server umount lustre-MDT0001 complete [ 2136.748168] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 2137.880469] Key type lgssc unregistered [ 2138.017342] LNet: 118750:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.020375] LNetError: 118750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.031413] LNet: Removed LNI 192.168.202.144@tcp [ 2138.339558] Key type .llcrypt unregistered [ 2138.340807] Key type ._llcrypt unregistered [ 2149.051712] Key type ._llcrypt registered [ 2149.054332] Key type .llcrypt registered [ 2149.098469] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2149.437461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2149.458089] alg: No test for adler32 (adler32-zlib) [ 2150.323650] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 2150.412445] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 2151.992134] Key type lgssc registered [ 2152.338744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2155.115842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2155.124158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.282294] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2156.286485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2156.290368] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.44/24@tcp [ 2157.609668] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2160.380533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.485402] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2160.494125] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.44/24@tcp [ 2160.506039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2161.751208] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2163.490203] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2164.697616] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2166.886663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2167.017001] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2168.989596] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2170.801285] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2181.351687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2186.721790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2186.722766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.724179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2186.730835] Lustre: Skipped 1 previous similar message [ 2190.805707] Lustre: server umount lustre-OST0000 complete [ 2196.448848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2196.451422] LustreError: Skipped 1 previous similar message [ 2196.452977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.456889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2196.458949] Lustre: Skipped 1 previous similar message [ 2198.230554] Lustre: server umount lustre-MDT0000 complete [ 2199.487445] LustreError: 122796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886235 with bad export cookie 820997657720820639 [ 2199.490782] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2199.492291] LustreError: 122796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2199.608040] Lustre: server umount lustre-MDT0001 complete [ 2203.122947] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 2204.196186] Key type lgssc unregistered [ 2204.322400] LNet: 123530:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.325300] LNetError: 123530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2204.333292] LNet: Removed LNI 192.168.202.144@tcp [ 2204.636946] Key type .llcrypt unregistered [ 2204.638160] Key type ._llcrypt unregistered [ 2213.461334] Key type ._llcrypt registered [ 2213.462708] Key type .llcrypt registered [ 2213.497542] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 2219.978592] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2220.360262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2220.403523] alg: No test for adler32 (adler32-zlib) [ 2221.270035] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 2221.352201] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 2222.944169] Key type lgssc registered [ 2223.297251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2235.458251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2235.463903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.558944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2236.571841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2236.608200] Lustre: lustre-MDT0000: new disk, initializing [ 2236.640815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.649222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2238.121746] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2242.484239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.522632] Lustre: 126868:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2242.544584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2242.546586] Lustre: Skipped 1 previous similar message [ 2242.579689] Lustre: lustre-MDT0001: new disk, initializing [ 2242.613437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2242.626122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2242.630543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2243.968446] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2247.147719] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2248.336578] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2250.727917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2250.826126] Lustre: lustre-OST0000: new disk, initializing [ 2250.828087] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2250.859349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.719127] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2253.891193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2253.894902] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2253.919932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2255.849317] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2257.228140] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2257.303500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2258.630530] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2258.702910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2264.037097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2264.037844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.039436] LustreError: Skipped 1 previous similar message [ 2264.039497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2264.044086] Lustre: Skipped 1 previous similar message [ 2265.434428] Lustre: server umount lustre-OST0000 complete [ 2268.128959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2268.132242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.137503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2268.139248] Lustre: Skipped 1 previous similar message [ 2269.153511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.154178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.857195] Lustre: server umount lustre-MDT0000 complete [ 2274.344605] LustreError: 126861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886310 with bad export cookie 17772053889371392624 [ 2274.347543] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2274.349279] LustreError: 126861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2274.489939] Lustre: server umount lustre-MDT0001 complete [ 2277.505683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.684868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.186453] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2282.125635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.258466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2283.565242] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2285.383628] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2286.667693] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2288.975658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2291.175521] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2293.188230] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2294.710274] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2302.975314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2304.568880] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2304.656166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2309.566490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2309.632225] Lustre: lustre-OST0001: new disk, initializing [ 2309.634363] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2309.669299] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2309.673868] Lustre: Skipped 1 previous similar message [ 2311.691806] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2315.102159] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2316.803149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2316.806568] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2316.829799] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2317.210113] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2318.638651] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2318.715932] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2319.995788] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0fb8708a800.ost_server_uuid 50 [ 2320.549370] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa0fb8708a800.ost_server_uuid in FULL state after 0 sec [ 2327.011713] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2327.012663] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2327.013505] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2327.013509] Lustre: Skipped 1 previous similar message [ 2327.016830] Lustre: Skipped 1 previous similar message [ 2327.019502] LustreError: Skipped 1 previous similar message [ 2328.583872] Lustre: server umount lustre-OST0001 complete [ 2332.131105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.131766] LustreError: 132738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.132038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2332.132042] Lustre: Skipped 1 previous similar message [ 2332.134672] Lustre: Skipped 1 previous similar message [ 2332.147501] LustreError: 132738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2336.081897] Lustre: server umount lustre-OST0000 complete [ 2338.785327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2342.369912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2342.370394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.372613] Lustre: Skipped 2 previous similar messages [ 2342.381758] Lustre: Skipped 1 previous similar message [ 2343.465265] Lustre: server umount lustre-MDT0000 complete [ 2344.871318] LustreError: 130656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886380 with bad export cookie 17772053889371393492 [ 2344.874487] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2344.875747] LustreError: 130656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2345.029833] Lustre: server umount lustre-MDT0001 complete [ 2352.373742] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 2353.461528] Key type lgssc unregistered [ 2353.583372] LNet: 136579:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2353.585539] LNetError: 136579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2353.592417] LNet: Removed LNI 192.168.202.144@tcp [ 2353.873401] Key type .llcrypt unregistered [ 2353.874835] Key type ._llcrypt unregistered [ 2367.048870] Key type ._llcrypt registered [ 2367.050151] Key type .llcrypt registered [ 2367.089750] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 2373.247445] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2373.655870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2373.670396] alg: No test for adler32 (adler32-zlib) [ 2374.525299] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 2374.600247] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 2376.176145] Key type lgssc registered [ 2376.529403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2388.129718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2388.139353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.226358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2389.237617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2389.270075] Lustre: lustre-MDT0000: new disk, initializing [ 2389.294281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.300388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2390.561867] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2394.873053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.918147] Lustre: 140670:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2394.946041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2394.947936] Lustre: Skipped 1 previous similar message [ 2394.988482] Lustre: lustre-MDT0001: new disk, initializing [ 2395.031788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2395.046656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2395.050970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2396.511265] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2399.783144] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2401.056729] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2403.478526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2403.608203] Lustre: lustre-OST0000: new disk, initializing [ 2403.611221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2403.649359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.416915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2404.422494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2404.465658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2405.679779] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2408.770394] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2410.761685] Lustre: Failing over lustre-OST0000 [ 2410.833522] Lustre: server umount lustre-OST0000 complete [ 2423.795177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2423.903892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.908951] Lustre: Skipped 1 previous similar message [ 2423.912930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.919423] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2425.215870] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2425.924212] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2427.281558] Lustre: Failing over lustre-MDT0000 [ 2427.377845] Lustre: server umount lustre-MDT0000 complete [ 2429.411611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.417172] Lustre: Skipped 2 previous similar messages [ 2430.368129] Lustre: 138897:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775886450/real 1775886450] req@0000000093bbe44c x1862151868279296/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1775886466 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2434.529234] LustreError: 140680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.535698] LustreError: 140680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2435.450873] LustreError: 140677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.488520] Lustre: 138897:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775886455/real 1775886455] req@000000004263070d x1862151868281344/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775886471 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2435.500630] Lustre: 138897:0:(client.c:2482:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2439.649628] LustreError: 141455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2439.655678] LustreError: 141455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2440.532302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2440.578217] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2440.582616] Lustre: 140672:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.202.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2440.663562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2440.685962] Lustre: lustre-OST0000: Recovery over after 0:15, of 3 clients 3 recovered and 0 were evicted. [ 2440.687423] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2440.692977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2440.727986] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2441.954805] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2445.687872] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2445.801131] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2445.804353] Lustre: Skipped 1 previous similar message [ 2445.807898] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2445.825264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2450.918214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2450.919665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.921246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2450.932073] Lustre: Skipped 1 previous similar message [ 2453.017682] Lustre: server umount lustre-OST0000 complete [ 2456.035469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2456.035680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.038453] LustreError: Skipped 1 previous similar message [ 2456.043463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2456.046381] Lustre: Skipped 2 previous similar messages [ 2460.512698] Lustre: server umount lustre-MDT0000 complete [ 2461.153108] LustreError: 140681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.160194] LustreError: 140681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2461.842548] LustreError: 140660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886497 with bad export cookie 14219589392477213620 [ 2461.845344] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2461.847048] LustreError: 140660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2461.965681] Lustre: server umount lustre-MDT0001 complete [ 2465.796283] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 2466.907857] Key type lgssc unregistered [ 2467.047543] LNet: 145893:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2467.050556] LNetError: 145893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.058322] LNet: Removed LNI 192.168.202.144@tcp [ 2467.371356] Key type .llcrypt unregistered [ 2467.372296] Key type ._llcrypt unregistered [ 2488.626054] Key type ._llcrypt registered [ 2488.627569] Key type .llcrypt registered [ 2488.674274] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2489.097426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2489.140584] alg: No test for adler32 (adler32-zlib) [ 2490.003742] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 2490.085724] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 2491.664286] Key type lgssc registered [ 2492.098262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2494.772108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2494.778872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.932828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.367719] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2500.601871] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2502.645237] LustreError: 147423:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2502.649098] LustreError: 147423:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2502.653202] LustreError: 147423:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2502.776897] Lustre: server umount lustre-MDT0000 complete [ 2508.143403] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 2511.716466] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2524.647956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.759239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2524.777941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2524.818104] Lustre: lustre-MDT0000: new disk, initializing [ 2524.844848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.853745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2526.275290] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2530.591202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.627056] Lustre: 150576:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2530.648610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2530.650660] Lustre: Skipped 1 previous similar message [ 2530.683560] Lustre: lustre-MDT0001: new disk, initializing [ 2530.714335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2530.726472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2530.730236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2531.939107] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2535.270116] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2536.507271] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2538.929961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2539.034537] Lustre: lustre-OST0000: new disk, initializing [ 2539.037537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2539.068621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2541.027884] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2544.173352] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2547.712312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2547.716407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2547.746671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2551.350459] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2551.930158] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:49:47 (1775886587) [ 2634.720881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2634.721144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2634.730188] Lustre: Skipped 2 previous similar messages [ 2634.739102] Lustre: Skipped 2 previous similar messages [ 2639.840670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2639.845215] Lustre: Skipped 2 previous similar messages [ 2644.960508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2644.963422] Lustre: Skipped 2 previous similar messages [ 2649.056116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2649.122278] Lustre: server umount lustre-MDT0000 complete [ 2650.080403] LustreError: 150582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.085883] LustreError: 150582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2650.333611] LustreError: 150568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886686 with bad export cookie 15382973911335588627 [ 2650.334661] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2650.338612] LustreError: 150568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2655.200779] LustreError: 150583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.200815] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.211206] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2656.486297] Lustre: server umount lustre-MDT0001 complete [ 2664.021766] Lustre: server umount lustre-OST0000 complete [ 2667.368662] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 2670.856460] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2683.986213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2684.096410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2684.110308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2684.154374] Lustre: lustre-MDT0000: new disk, initializing [ 2684.181677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2684.187635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2685.707605] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2689.959177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.999987] Lustre: 156797:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2690.024160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2690.025894] Lustre: Skipped 1 previous similar message [ 2690.058533] Lustre: lustre-MDT0001: new disk, initializing [ 2690.102106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2690.106219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2691.481515] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2694.668459] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2695.885859] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2698.138556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2698.260199] Lustre: lustre-OST0000: new disk, initializing [ 2698.261922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2698.290481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.292496] Lustre: Skipped 1 previous similar message [ 2700.255259] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2701.496607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2701.500121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2701.531180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2703.269275] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2704.692334] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2704.765451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2706.103572] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2706.179074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2706.913490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2706.915107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2706.916627] LustreError: Skipped 1 previous similar message [ 2706.916819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2706.922196] Lustre: Skipped 1 previous similar message [ 2717.154860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2717.157176] Lustre: Skipped 4 previous similar messages [ 2721.248136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2721.301586] Lustre: server umount lustre-OST0000 complete [ 2725.856887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2725.859930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.673257] Lustre: server umount lustre-MDT0000 complete [ 2730.121495] LustreError: 156790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886765 with bad export cookie 15382973911335747443 [ 2730.126675] LustreError: 156790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2730.126727] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2730.248542] Lustre: server umount lustre-MDT0001 complete [ 2733.387512] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:52:48 (1775886768) [ 2739.748223] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 2743.189318] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2756.457222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2756.564558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2756.584667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2756.625811] Lustre: lustre-MDT0000: new disk, initializing [ 2756.657495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2756.665413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2758.134178] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2762.613644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.660394] Lustre: 163109:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2762.768085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2764.419800] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2767.562114] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2768.840259] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2771.312210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.816392] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2777.054751] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2778.624484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2778.665335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2788.835455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2788.835519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.836166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2788.836171] Lustre: Skipped 3 previous similar messages [ 2788.837820] LustreError: Skipped 1 previous similar message [ 2788.846330] Lustre: Skipped 2 previous similar messages [ 2790.550110] Lustre: server umount lustre-OST0000 complete [ 2793.441155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2806.240271] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2806.312265] Lustre: server umount lustre-MDT0000 complete [ 2807.639634] LustreError: 163100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886843 with bad export cookie 15382973911335748311 [ 2807.639939] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2807.644883] LustreError: 163100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2807.793541] Lustre: server umount lustre-MDT0001 complete [ 2810.618409] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:54:05 (1775886845) [ 2815.741682] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 2818.853539] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2830.716242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2830.802812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2830.813245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2830.814767] Lustre: Skipped 3 previous similar messages [ 2830.849750] Lustre: lustre-MDT0000: new disk, initializing [ 2830.851316] Lustre: Skipped 2 previous similar messages [ 2830.874517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2830.876913] Lustre: Skipped 2 previous similar messages [ 2830.882918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2830.885398] Lustre: Skipped 2 previous similar messages [ 2832.077448] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2836.053736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.095436] Lustre: 169027:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2836.196726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2837.450934] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2840.492363] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2841.605297] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2843.740721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2845.618833] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2848.584236] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2849.302904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2859.490956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2859.491495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2859.492092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2859.492096] Lustre: Skipped 7 previous similar messages [ 2859.493161] LustreError: Skipped 1 previous similar message [ 2859.497239] Lustre: Skipped 3 previous similar messages [ 2861.520993] Lustre: server umount lustre-OST0000 complete [ 2868.830676] Lustre: server umount lustre-MDT0000 complete [ 2869.729099] LustreError: 169037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.734673] LustreError: 169037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2870.045933] LustreError: 169019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886905 with bad export cookie 15382973911335749396 [ 2870.048697] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2870.050369] LustreError: 169019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2870.167601] Lustre: server umount lustre-MDT0001 complete [ 2873.749583] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 2874.762092] Key type lgssc unregistered [ 2874.888376] LNet: 172501:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2874.892156] LNetError: 172501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2874.901433] LNet: Removed LNI 192.168.202.144@tcp [ 2875.189215] Key type .llcrypt unregistered [ 2875.190596] Key type ._llcrypt unregistered [ 2883.691143] Key type ._llcrypt registered [ 2883.692359] Key type .llcrypt registered [ 2883.724757] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2883.983054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2883.999728] alg: No test for adler32 (adler32-zlib) [ 2884.849823] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 2884.926205] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 2886.504104] Key type lgssc registered [ 2886.842494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2889.214660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2889.219647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.373323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.568356] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2894.248952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.374880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2895.595288] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2897.257460] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2898.376418] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2900.547259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2900.659239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2902.431545] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2904.160481] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2916.008594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2931.682221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2931.682952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.684939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2931.693444] Lustre: Skipped 1 previous similar message [ 2936.804692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2936.806934] Lustre: Skipped 2 previous similar messages [ 2937.752961] Lustre: server umount lustre-OST0000 complete [ 2940.384867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2940.388342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.393293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2941.922142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2945.246273] Lustre: server umount lustre-MDT0000 complete [ 2946.561292] LustreError: 176143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886982 with bad export cookie 4545244956534383694 [ 2946.565538] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2946.566488] LustreError: 176143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2946.703600] Lustre: server umount lustre-MDT0001 complete [ 2950.421807] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 2951.547433] Key type lgssc unregistered [ 2951.677435] LNet: 177078:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2951.680865] LNetError: 177078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2951.691377] LNet: Removed LNI 192.168.202.144@tcp [ 2952.024342] Key type .llcrypt unregistered [ 2952.026102] Key type ._llcrypt unregistered [ 2961.397500] Key type ._llcrypt registered [ 2961.398955] Key type .llcrypt registered [ 2961.444050] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 2961.788954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2961.808663] alg: No test for adler32 (adler32-zlib) [ 2962.668384] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 2962.754796] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 2964.336173] Key type lgssc registered [ 2964.735732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2967.735324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2967.744950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2968.938533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2970.411488] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2973.454661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.588697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2975.005221] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2976.980919] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2978.254779] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2980.597729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2980.728648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.767955] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 2984.682900] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2994.026809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2999.267696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2999.269959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.271879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2999.274476] Lustre: Skipped 1 previous similar message [ 3000.189360] Lustre: lustre-OST0000: Not available for connect from 192.168.202.44@tcp (stopping) [ 3000.191377] Lustre: Skipped 1 previous similar message [ 3003.474868] Lustre: server umount lustre-OST0000 complete [ 3005.250574] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffffa0fb875ae800.ost_server_uuid 50 [ 3005.796354] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa0fb875ae800.ost_server_uuid in DISCONN state after 0 sec [ 3006.412431] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3016.506848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3016.507337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3016.510386] Lustre: Skipped 1 previous similar message [ 3019.617453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3019.620246] Lustre: Skipped 2 previous similar messages [ 3022.681193] Lustre: server umount lustre-MDT0000 complete [ 3023.930238] LustreError: 178433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887059 with bad export cookie 14180698888533513659 [ 3023.933468] LustreError: 178433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3023.933688] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3024.054159] Lustre: server umount lustre-MDT0001 complete [ 3026.837466] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:57:42 (1775887062) [ 3029.241500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.401435] LustreError: 181981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3029.407996] LustreError: 181981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3029.423559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3030.758367] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3033.628886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3033.746694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3035.143825] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3037.064075] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3038.371476] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3040.709075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3042.719291] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3044.608557] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3047.176637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3047.263181] Lustre: lustre-OST0001: new disk, initializing [ 3047.268158] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3047.304324] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3047.307269] Lustre: Skipped 1 previous similar message [ 3049.234289] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3049.731888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3049.736176] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3049.770326] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3052.409848] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3054.048912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3054.051019] LustreError: Skipped 1 previous similar message [ 3054.052530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3054.058506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3059.452536] Lustre: server umount lustre-OST0000 complete [ 3060.196066] LustreError: 183572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.200338] LustreError: 183572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3061.587367] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3061.661986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3062.937244] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3063.004907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3063.160138] LustreError: 183575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.316826] LustreError: 184903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.321762] LustreError: 184903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3070.436654] LustreError: 183574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.442408] LustreError: 183574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3075.556037] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3075.556393] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3075.558816] LustreError: Skipped 1 previous similar message [ 3075.563596] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3075.568180] Lustre: Skipped 4 previous similar messages [ 3079.254671] Lustre: server umount lustre-OST0001 complete [ 3080.673353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.677934] Lustre: Skipped 1 previous similar message [ 3085.280838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3086.683925] Lustre: server umount lustre-MDT0000 complete [ 3087.914056] LustreError: 185792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887123 with bad export cookie 14180698888533514716 [ 3087.917161] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3087.917278] LustreError: 185792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3088.028979] Lustre: server umount lustre-MDT0001 complete [ 3095.508075] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:58:50 (1775887130) [ 3097.886222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3098.042316] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3098.055258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3098.145496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.561079] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3102.718765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3102.753173] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3102.770102] Lustre: 187750:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3104.316473] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3106.297820] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3107.609396] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3109.941531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.030795] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3112.119755] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3112.662460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3114.053402] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3116.294545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3116.318344] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3118.370271] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3120.134797] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3121.635851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3121.639695] Lustre: Skipped 1 previous similar message [ 3121.641577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3121.643056] Lustre: Skipped 4 previous similar messages [ 3121.674511] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3127.036323] Lustre: server umount lustre-OST0000 complete [ 3130.744206] LustreError: 189373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.749896] LustreError: 189373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3142.119256] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3142.120343] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.121939] LustreError: Skipped 1 previous similar message [ 3142.127864] Lustre: Skipped 2 previous similar messages [ 3143.831352] Lustre: server umount lustre-OST0001 complete [ 3151.197480] Lustre: server umount lustre-MDT0000 complete [ 3152.352933] LustreError: 187759:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.357748] LustreError: 187759:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3152.362911] LustreError: 187741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887188 with bad export cookie 14180698888533515878 [ 3152.364716] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3152.365758] LustreError: 187741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3152.467929] Lustre: server umount lustre-MDT0001 complete [ 3160.096127] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:59:55 (1775887195) [ 3165.199713] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 3168.366091] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 3180.402747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.495140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3180.506089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3180.548164] Lustre: lustre-MDT0000: new disk, initializing [ 3180.575778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.577698] Lustre: Skipped 3 previous similar messages [ 3180.583389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3181.901021] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3185.984545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.026329] Lustre: 195317:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3186.048629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3186.050348] Lustre: Skipped 1 previous similar message [ 3186.085679] Lustre: lustre-MDT0001: new disk, initializing [ 3186.126774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3186.130178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3187.420952] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3190.571223] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3191.780305] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3194.154685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3194.257301] Lustre: lustre-OST0000: new disk, initializing [ 3194.259331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3195.640650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3195.643827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3195.678066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3196.170335] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3199.355505] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3200.828934] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3200.906517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3202.260852] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3202.339456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3206.114298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3206.116164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.117979] LustreError: Skipped 1 previous similar message [ 3206.124268] Lustre: Skipped 2 previous similar messages [ 3206.127493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3206.130300] Lustre: Skipped 8 previous similar messages [ 3217.376118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3217.429946] Lustre: server umount lustre-OST0000 complete [ 3233.248074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3233.309188] Lustre: server umount lustre-MDT0000 complete [ 3234.574457] LustreError: 196095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887270 with bad export cookie 14180698888533517614 [ 3234.578316] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3234.579059] LustreError: 196095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3234.717531] Lustre: server umount lustre-MDT0001 complete [ 3237.653686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3237.840625] LustreError: 199142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3239.188933] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3241.963478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3243.306927] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3245.038126] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3246.223490] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3248.498370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3248.609854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3248.611717] Lustre: Skipped 4 previous similar messages [ 3250.523166] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3252.343454] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3253.710710] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3253.780319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3255.162919] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3255.237284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3256.802228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3256.806112] LustreError: Skipped 1 previous similar message [ 3257.936292] Lustre: server umount lustre-OST0000 complete [ 3259.847887] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3259.935421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3261.298318] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3261.382389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3285.070905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3287.046798] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3289.014591] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3289.604699] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3292.694742] Lustre: server umount lustre-OST0000 complete [ 3294.028730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.029553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3294.032595] Lustre: Skipped 5 previous similar messages [ 3294.035761] Lustre: Skipped 12 previous similar messages [ 3296.157580] Lustre: server umount lustre-MDT0000 complete [ 3297.511331] LustreError: 200758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887333 with bad export cookie 14180698888533518482 [ 3297.514525] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.517398] LustreError: 200758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3297.642327] Lustre: server umount lustre-MDT0001 complete [ 3300.612375] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:02:15 (1775887335) [ 3302.738257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.878292] LustreError: 204616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3302.885755] LustreError: 204616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3304.147852] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3306.963409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.392819] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3310.178252] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3311.423225] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3313.683199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3315.871123] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3317.688324] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3319.062189] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3328.318225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3329.579927] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3329.649293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3331.832766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3331.923107] Lustre: lustre-OST0001: new disk, initializing [ 3331.924920] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3333.851443] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3336.781637] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3338.113388] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3340.799243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3340.801962] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3340.826889] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3341.251904] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3342.515915] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3342.585243] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3345.892410] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3349.243249] Lustre: server umount lustre-OST0001 complete [ 3351.062763] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3351.135743] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3352.379829] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3352.457549] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3375.941907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3377.897664] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3379.541026] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3380.037390] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3386.742883] Lustre: server umount lustre-OST0001 complete [ 3404.882135] Lustre: server umount lustre-OST0000 complete [ 3412.319552] Lustre: server umount lustre-MDT0000 complete [ 3413.596231] LustreError: 206233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887449 with bad export cookie 14180698888533519399 [ 3413.599603] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3413.600175] LustreError: 206233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3419.751868] Lustre: server umount lustre-MDT0001 complete [ 3427.272177] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:04:22 (1775887462) [ 3429.510962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3429.618637] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3429.630119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3429.704976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3429.707340] Lustre: Skipped 6 previous similar messages [ 3430.992527] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3433.913985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.947251] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3433.956663] Lustre: 212640:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3435.426323] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3437.270546] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3438.495755] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3440.886855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3440.981375] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3443.089488] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3444.939588] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3453.747482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3453.770138] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3455.758313] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3457.499234] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3460.021739] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3461.294194] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3461.378361] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3462.698202] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3462.770685] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3463.937396] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0fb86d0c800.ost_server_uuid 50 [ 3464.435336] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa0fb86d0c800.ost_server_uuid in FULL state after 0 sec [ 3464.806178] Lustre: Permanently deactivating lustre-OST0001 [ 3464.811184] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3465.658316] Lustre: Permanently reactivating lustre-OST0001 [ 3470.311930] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3470.313391] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3470.314324] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3470.314330] Lustre: Skipped 18 previous similar messages [ 3470.314526] LustreError: Skipped 4 previous similar messages [ 3470.326295] Lustre: Skipped 7 previous similar messages [ 3472.698685] Lustre: server umount lustre-OST0001 complete [ 3475.429080] LustreError: 215365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.434025] LustreError: 215365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 3480.148996] Lustre: server umount lustre-OST0000 complete [ 3487.585751] Lustre: server umount lustre-MDT0000 complete [ 3488.868265] LustreError: 212674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887524 with bad export cookie 14180698888533520659 [ 3488.872316] LustreError: 212674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3488.873402] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3488.997760] Lustre: server umount lustre-MDT0001 complete [ 3496.356269] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:05:31 (1775887531) [ 3498.826749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3498.917563] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3498.927308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3498.930310] Lustre: Skipped 5 previous similar messages [ 3500.238990] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3503.071953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3503.109637] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3503.117824] Lustre: 218741:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3504.640023] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3506.457429] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3507.673515] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3509.999743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3510.081837] Lustre: Permanently deactivating lustre-OST0000 [ 3510.086546] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3511.982929] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3513.648150] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3515.735644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3515.825506] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3515.869525] LustreError: 221299:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 3515.878267] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3517.520368] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3519.189045] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3519.730499] Lustre: Permanently reactivating lustre-OST0000 [ 3524.603115] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3524.606269] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3524.609787] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3524.624483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3524.631331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3533.636644] Lustre: server umount lustre-OST0001 complete [ 3549.152177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3549.208314] Lustre: server umount lustre-OST0000 complete [ 3556.763908] Lustre: server umount lustre-MDT0000 complete [ 3558.042448] LustreError: 218732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887593 with bad export cookie 14180698888533522290 [ 3558.045773] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3558.047161] LustreError: 218732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3558.169050] Lustre: server umount lustre-MDT0001 complete [ 3561.445363] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 3562.507412] Key type lgssc unregistered [ 3562.645496] LNet: 223124:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3562.649171] LNetError: 223124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3562.655445] LNet: Removed LNI 192.168.202.144@tcp [ 3562.965462] Key type .llcrypt unregistered [ 3562.966718] Key type ._llcrypt unregistered [ 3571.398759] Key type ._llcrypt registered [ 3571.400656] Key type .llcrypt registered [ 3571.441891] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 3571.881111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3571.921106] alg: No test for adler32 (adler32-zlib) [ 3572.769543] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3572.857504] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 3574.440197] Key type lgssc registered [ 3574.844052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3578.213283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3578.220226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3579.401354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.739281] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3583.708169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3583.752371] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3583.764421] Lustre: 224555:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3583.771277] Lustre: Permanently deactivating lustre-MDT0001 [ 3583.779486] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3583.901753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3585.006354] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3585.008931] LustreError: 224596:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3585.012606] LustreError: 224596:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3585.016600] LustreError: 224596:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 1, failed: rc = -108 [ 3585.231295] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3587.067414] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3588.354973] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3590.708320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3590.832513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.808567] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3593.892557] LustreError: 226181:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3593.895429] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3593.898903] LustreError: 226181:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3594.764170] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3597.111113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3597.182053] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3599.184828] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3601.013117] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3601.200485] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3604.266350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3604.266471] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3606.806623] Lustre: Permanently reactivating lustre-MDT0001 [ 3606.812948] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3606.815179] Lustre: Skipped 2 previous similar messages [ 3607.421043] Lustre: lustre-MDT0001: Client 02f26d26-9e56-42a5-adf0-76d4ce9e13f0 (at 192.168.202.44@tcp) reconnecting [ 3609.087430] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3609.595599] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3611.267885] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3611.344046] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3612.670105] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3612.740110] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3613.571607] Lustre: Permanently deactivating lustre-MDT0001 [ 3613.577739] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3613.581082] Lustre: Skipped 2 previous similar messages [ 3616.702219] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3624.929639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3624.931632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.935064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3625.954352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3625.956489] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3625.961728] Lustre: Skipped 2 previous similar messages [ 3629.403397] Lustre: server umount lustre-MDT0000 complete [ 3630.690464] LustreError: 224547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887666 with bad export cookie 2253176797358481236 [ 3630.693581] LustreError: 224547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3630.694211] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3630.808561] Lustre: server umount lustre-MDT0001 complete [ 3638.140541] Lustre: server umount lustre-OST0000 complete [ 3641.846664] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:07:57 (1775887677) [ 3645.481711] Lustre: server umount lustre-OST0001 complete [ 3647.656657] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 3651.011398] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 3662.928571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3663.018794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3663.022210] Lustre: Skipped 2 previous similar messages [ 3663.031133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3663.069288] Lustre: lustre-MDT0000: new disk, initializing [ 3663.092963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3663.100127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3664.396183] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3668.476237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3668.515639] Lustre: 232201:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3668.536592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3668.538514] Lustre: Skipped 1 previous similar message [ 3668.574325] Lustre: lustre-MDT0001: new disk, initializing [ 3668.615226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3668.618506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3669.876224] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3673.017688] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3674.171581] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3676.372890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3676.477920] Lustre: lustre-OST0000: new disk, initializing [ 3676.479786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3676.505575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3676.507489] Lustre: Skipped 1 previous similar message [ 3678.306663] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3681.397263] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3681.787145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3681.790373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3681.821880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3687.271766] LustreError: 232208:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3691.502484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3691.578415] Lustre: lustre-OST0001: new disk, initializing [ 3691.580985] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3693.634326] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3696.654348] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3697.360138] LustreError: 232208:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3697.366905] LustreError: 232209:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3698.172687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3698.176556] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3698.204649] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3699.166338] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3707.456103] LustreError: 232209:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3708.286158] Lustre: lustre-OST0001: Not available for connect from 192.168.202.44@tcp (stopping) [ 3708.288239] Lustre: Skipped 3 previous similar messages [ 3708.388176] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3708.388849] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3708.390542] LustreError: Skipped 1 previous similar message [ 3708.395133] Lustre: Skipped 1 previous similar message [ 3713.400317] Lustre: lustre-OST0001: Not available for connect from 192.168.202.44@tcp (stopping) [ 3713.403112] Lustre: Skipped 2 previous similar messages [ 3714.176922] Lustre: server umount lustre-OST0001 complete [ 3718.627454] LustreError: 233924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3718.631655] LustreError: 233924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3723.747341] LustreError: 233922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3723.753455] LustreError: 233922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3727.841272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3727.841341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3727.843677] LustreError: Skipped 1 previous similar message [ 3727.844280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3727.847356] Lustre: Skipped 1 previous similar message [ 3727.851807] Lustre: Skipped 3 previous similar messages [ 3728.868544] LustreError: 233924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3728.873646] LustreError: 233924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3732.376336] Lustre: server umount lustre-OST0000 complete [ 3733.986270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3733.990558] Lustre: Skipped 1 previous similar message [ 3739.105163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3739.107438] Lustre: Skipped 5 previous similar messages [ 3739.952295] Lustre: server umount lustre-MDT0000 complete [ 3741.232358] LustreError: 232977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887777 with bad export cookie 2253176797358484050 [ 3741.235589] LustreError: 232977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3741.238442] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3747.441957] Lustre: server umount lustre-MDT0001 complete [ 3751.166334] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 3752.226684] Key type lgssc unregistered [ 3752.351295] LNet: 236853:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3752.353842] LNetError: 236853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3752.362299] LNet: Removed LNI 192.168.202.144@tcp [ 3752.661609] Key type .llcrypt unregistered [ 3752.662817] Key type ._llcrypt unregistered [ 3766.858428] Key type ._llcrypt registered [ 3766.859478] Key type .llcrypt registered [ 3766.894996] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 3767.336925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3767.353440] alg: No test for adler32 (adler32-zlib) [ 3768.203458] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3768.280932] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 3769.856149] Key type lgssc registered [ 3770.210860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3772.696244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3772.702873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3773.792727] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3773.800840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3773.860582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3775.149346] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3778.029308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3778.062121] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3778.072296] Lustre: 239020:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3778.164132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3779.456369] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3781.248868] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3782.525911] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3784.846413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3784.930848] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3784.992396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3786.862607] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3788.602625] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3792.911355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3833.826988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3833.829277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.830709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3833.833195] Lustre: Skipped 1 previous similar message [ 3838.502472] Lustre: server umount lustre-OST0000 complete [ 3839.978330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3844.094971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3844.224443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3846.133422] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3848.000780] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3853.478320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3853.481932] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3853.482020] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3853.487126] Lustre: Skipped 1 previous similar message [ 3884.515914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3884.516034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3884.519686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3884.520934] LustreError: Skipped 1 previous similar message [ 3884.523689] Lustre: Skipped 2 previous similar messages [ 3887.131510] Lustre: server umount lustre-OST0000 complete [ 3889.634839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3889.635717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3889.639214] Lustre: Skipped 2 previous similar messages [ 3889.642040] Lustre: Skipped 1 previous similar message [ 3894.486662] Lustre: server umount lustre-MDT0000 complete [ 3895.775136] LustreError: 239010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887931 with bad export cookie 14503412055081195428 [ 3895.778747] LustreError: 239010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3895.778917] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3895.895701] Lustre: server umount lustre-MDT0001 complete [ 3899.498490] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 3900.524106] Key type lgssc unregistered [ 3900.642328] LNet: 243970:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3900.644564] LNetError: 243970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3900.653342] LNet: Removed LNI 192.168.202.144@tcp [ 3900.948436] Key type .llcrypt unregistered [ 3900.949658] Key type ._llcrypt unregistered [ 3909.881353] Key type ._llcrypt registered [ 3909.882456] Key type .llcrypt registered [ 3909.923910] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 3910.303947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3910.312671] alg: No test for adler32 (adler32-zlib) [ 3911.182190] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3911.271715] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 3912.856163] Key type lgssc registered [ 3913.282875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3915.843750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3915.850901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3917.010593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.281844] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3921.144252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3921.272536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3922.608918] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3924.437967] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3925.648111] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3927.906938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3928.020217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3929.866935] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3931.625820] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3941.351221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3951.586653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3951.587969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3951.589220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3951.592692] Lustre: Skipped 1 previous similar message [ 3951.595968] Lustre: Skipped 1 previous similar message [ 3955.735061] Lustre: server umount lustre-OST0000 complete [ 3957.217116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3957.219704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3957.223972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3961.825356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3961.825950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3963.160489] Lustre: server umount lustre-MDT0000 complete [ 3964.358073] LustreError: 245327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888000 with bad export cookie 5580022155509985586 [ 3964.361218] LustreError: 245327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3964.361590] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3964.477072] Lustre: server umount lustre-MDT0001 complete [ 3967.949802] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 3968.967204] Key type lgssc unregistered [ 3969.097574] LNet: 249315:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3969.100378] LNetError: 249315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3969.110471] LNet: Removed LNI 192.168.202.144@tcp [ 3969.414158] Key type .llcrypt unregistered [ 3969.415212] Key type ._llcrypt unregistered [ 3976.245257] Key type ._llcrypt registered [ 3976.246263] Key type .llcrypt registered [ 3976.290738] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 3976.663665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3976.669933] alg: No test for adler32 (adler32-zlib) [ 3977.541588] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3977.635898] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 3979.224195] Key type lgssc registered [ 3979.628687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3982.333464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3982.340258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.526413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3984.866468] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3987.702469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3987.817570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3989.131353] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3990.955314] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3992.152490] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3994.379061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3994.498085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3996.440509] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 3998.215731] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4008.870722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4018.145182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4018.147465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.151456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4019.170584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4019.170777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.171702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4023.571441] Lustre: server umount lustre-OST0000 complete [ 4028.897175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4028.901471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4028.906970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4028.909165] Lustre: Skipped 1 previous similar message [ 4034.529062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4034.531581] Lustre: Skipped 3 previous similar messages [ 4039.136100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4039.196573] Lustre: server umount lustre-MDT0000 complete [ 4039.650144] LustreError: 250497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4039.655864] LustreError: 250497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4040.433391] LustreError: 252818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888076 with bad export cookie 14174356283064728333 [ 4040.436724] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4040.437801] LustreError: 252818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4040.547873] Lustre: server umount lustre-MDT0001 complete [ 4044.282937] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 4045.302732] Key type lgssc unregistered [ 4045.428303] LNet: 253841:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4045.431817] LNetError: 253841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4045.441270] LNet: Removed LNI 192.168.202.144@tcp [ 4045.708018] Key type .llcrypt unregistered [ 4045.709172] Key type ._llcrypt unregistered [ 4054.745039] Key type ._llcrypt registered [ 4054.746064] Key type .llcrypt registered [ 4054.780283] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 4055.127665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4055.163807] alg: No test for adler32 (adler32-zlib) [ 4056.013549] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 4056.094242] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 4057.672170] Key type lgssc registered [ 4058.016465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4060.675860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4060.686720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4061.858539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4063.149811] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4066.014856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.142937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4067.424975] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4069.142422] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4070.314871] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4072.520126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4072.630866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4074.599560] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4076.431356] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4086.951102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4096.480807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4096.483664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.488818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4097.507451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4097.507502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.507795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4100.688972] Lustre: server umount lustre-OST0000 complete [ 4102.112964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4102.115641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4102.119874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4102.122418] Lustre: Skipped 1 previous similar message [ 4107.745824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4107.747568] Lustre: Skipped 2 previous similar messages [ 4108.059870] Lustre: server umount lustre-MDT0000 complete [ 4109.301307] LustreError: 255198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888145 with bad export cookie 11060273465463473824 [ 4109.305739] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4109.306336] LustreError: 255198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4109.424773] Lustre: server umount lustre-MDT0001 complete [ 4113.052922] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 4114.035800] Key type lgssc unregistered [ 4114.160366] LNet: 259189:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4114.162791] LNetError: 259189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4114.171287] LNet: Removed LNI 192.168.202.144@tcp [ 4114.477424] Key type .llcrypt unregistered [ 4114.479506] Key type ._llcrypt unregistered [ 4121.775655] Key type ._llcrypt registered [ 4121.776570] Key type .llcrypt registered [ 4121.821481] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 4122.223936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4122.251691] alg: No test for adler32 (adler32-zlib) [ 4123.155716] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 4123.267648] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 4124.864134] Key type lgssc registered [ 4125.217926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4127.728707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4127.734945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4128.905436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4130.218679] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4132.992596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.117156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4134.423475] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4136.200968] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4137.415635] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4139.721059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4139.852891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4141.730070] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4143.491200] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4154.219146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4158.433125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4158.435156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4158.438702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4159.459461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4159.460515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4159.460655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4163.859364] Lustre: server umount lustre-OST0000 complete [ 4169.185068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4169.187812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4169.191751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4169.194502] Lustre: Skipped 1 previous similar message [ 4171.291364] Lustre: server umount lustre-MDT0000 complete [ 4172.534846] LustreError: 261127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888208 with bad export cookie 6080736379609528942 [ 4172.538239] LustreError: 261127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4172.539100] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4172.677053] Lustre: server umount lustre-MDT0001 complete [ 4176.274081] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 4177.389513] Key type lgssc unregistered [ 4177.529560] LNet: 263751:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4177.531742] LNetError: 263751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4177.539495] LNet: Removed LNI 192.168.202.144@tcp [ 4177.863146] Key type .llcrypt unregistered [ 4177.864263] Key type ._llcrypt unregistered [ 4181.437634] Key type ._llcrypt registered [ 4181.438624] Key type .llcrypt registered [ 4181.479067] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4187.044182] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 4193.114054] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 4193.499171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4193.516090] alg: No test for adler32 (adler32-zlib) [ 4194.366941] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 4194.447180] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 4196.024144] Key type lgssc registered [ 4196.369302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4208.032375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4208.041188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4209.128585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4209.144114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4209.191389] Lustre: lustre-MDT0000: new disk, initializing [ 4209.218201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4209.225224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4210.669661] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4215.057623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4215.099833] Lustre: 267222:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4215.129764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4215.131533] Lustre: Skipped 1 previous similar message [ 4215.172906] Lustre: lustre-MDT0001: new disk, initializing [ 4215.213375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4215.229557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4215.233065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4216.633775] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4219.799876] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4220.951380] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4223.250151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4223.363630] Lustre: lustre-OST0000: new disk, initializing [ 4223.366068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4223.399874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4225.151493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4225.157039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4225.203452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4225.339625] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4228.476830] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4229.803556] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4229.871420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4231.153174] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4231.224834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4235.750634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4235.751479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.755367] LustreError: Skipped 1 previous similar message [ 4235.764479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4235.770446] Lustre: Skipped 1 previous similar message [ 4237.973496] Lustre: server umount lustre-OST0000 complete [ 4240.867513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4240.867569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4240.868295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4240.882543] Lustre: Skipped 2 previous similar messages [ 4245.342229] Lustre: server umount lustre-MDT0000 complete [ 4246.616291] LustreError: 268003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888282 with bad export cookie 1162645783024713789 [ 4246.619282] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4246.621638] LustreError: 268003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4246.749647] Lustre: server umount lustre-MDT0001 complete [ 4249.660800] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:18:04 (1775888284) [ 4251.842910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4252.006995] LustreError: 271226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4252.013118] LustreError: 271226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4252.035531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4253.388084] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4256.227956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4256.356631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4257.704329] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4259.506358] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4260.685432] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4262.966194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4264.943268] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4266.736205] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4281.827504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.829331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4281.833241] Lustre: Skipped 1 previous similar message [ 4281.833567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4281.834905] Lustre: Skipped 2 previous similar messages [ 4286.096352] Lustre: server umount lustre-OST0000 complete [ 4290.018025] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4290.020547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4290.025520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4293.654560] Lustre: server umount lustre-MDT0000 complete [ 4294.942435] LustreError: 271945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888330 with bad export cookie 1162645783024714657 [ 4294.945789] LustreError: 271948:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4294.946733] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4295.073807] Lustre: server umount lustre-MDT0001 complete [ 4298.736119] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 4299.823540] Key type lgssc unregistered [ 4299.959385] LNet: 274427:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4299.962215] LNetError: 274427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4299.975283] LNet: Removed LNI 192.168.202.144@tcp [ 4300.267480] Key type .llcrypt unregistered [ 4300.268497] Key type ._llcrypt unregistered [ 4308.836758] Key type ._llcrypt registered [ 4308.838369] Key type .llcrypt registered [ 4308.880565] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 4314.658077] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 4315.130345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4315.151984] alg: No test for adler32 (adler32-zlib) [ 4316.018766] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 4316.108791] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 4317.696131] Key type lgssc registered [ 4318.096165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4332.803735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4332.809342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4333.918398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4333.934793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4333.974280] Lustre: lustre-MDT0000: new disk, initializing [ 4334.000592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4334.007148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4335.278897] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4339.455376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4339.496226] Lustre: 278188:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4339.520237] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4339.522840] Lustre: Skipped 1 previous similar message [ 4339.563486] Lustre: lustre-MDT0001: new disk, initializing [ 4339.599771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4339.611471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4339.615122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4340.934452] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4343.998185] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4345.154957] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4347.390062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4347.512174] Lustre: lustre-OST2710: new disk, initializing [ 4347.514158] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4347.551659] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4349.530351] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4351.287137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4351.292676] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4351.329401] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4352.602647] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4354.938739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4355.006338] Lustre: lustre-OST03e8: new disk, initializing [ 4355.008200] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4355.040099] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4356.996621] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4358.833309] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4358.988954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4358.993088] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4359.037368] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4361.411705] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4361.477882] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4362.768417] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4362.835225] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4364.125987] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4364.203876] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4365.508136] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4365.579832] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4369.379932] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4369.381632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4369.384102] Lustre: Skipped 3 previous similar messages [ 4369.386452] Lustre: Skipped 2 previous similar messages [ 4372.703855] Lustre: server umount lustre-MDT0000 complete [ 4373.921957] LustreError: 282433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888409 with bad export cookie 3210482419834572145 [ 4373.925038] LustreError: 282433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4373.925693] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4374.036120] Lustre: server umount lustre-MDT0001 complete [ 4385.487345] Lustre: server umount lustre-OST2710 complete [ 4397.031308] Lustre: server umount lustre-OST03e8 complete [ 4402.941032] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 4406.164556] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 4418.099222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.200149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4418.213142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4418.255604] Lustre: lustre-MDT0000: new disk, initializing [ 4418.283977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4418.290549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4419.651722] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4423.795519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4423.837717] Lustre: 285902:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4423.935571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4425.209422] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4428.464600] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4429.673478] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4431.957362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4432.064161] Lustre: lustre-OST0000: new disk, initializing [ 4432.065842] Lustre: Skipped 1 previous similar message [ 4432.067778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4432.070508] Lustre: Skipped 2 previous similar messages [ 4432.105179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.107703] Lustre: Skipped 1 previous similar message [ 4433.880519] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4437.011730] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4438.408314] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4439.048559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4439.053246] Lustre: Skipped 1 previous similar message [ 4439.056928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4439.090321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4439.538929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4440.833137] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4440.900543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4444.133067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4444.134553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4444.138881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4444.142914] Lustre: Skipped 1 previous similar message [ 4447.633423] Lustre: server umount lustre-OST0000 complete [ 4449.250256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.250880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4449.254292] Lustre: Skipped 2 previous similar messages [ 4449.258383] Lustre: Skipped 1 previous similar message [ 4454.369304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4454.373026] Lustre: Skipped 1 previous similar message [ 4459.489073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4459.491073] Lustre: Skipped 1 previous similar message [ 4463.072082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4463.140883] Lustre: server umount lustre-MDT0000 complete [ 4464.403221] LustreError: 285895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888500 with bad export cookie 3210482419834573811 [ 4464.406163] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4464.408519] LustreError: 285895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4464.538144] Lustre: server umount lustre-MDT0001 complete [ 4467.232812] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:21:42 (1775888502) [ 4467.749890] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4468.303934] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:21:43 (1775888503) [ 4469.313126] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 4478.389175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4478.483422] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4478.491188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4478.556648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.887047] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4482.748873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.780894] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4482.789861] Lustre: 291338:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4484.219823] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4485.960777] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4487.203878] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4502.499548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4502.587100] Lustre: Denying initial registration attempt from nid 192.168.202.144@tcp, specified as failover [ 4502.591480] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4502.595974] LustreError: 292948:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -99 [ 4502.599554] LustreError: 292948:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 4502.602633] LustreError: 292948:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 4502.635409] Lustre: server umount lustre-OST0000 complete [ 4502.637357] LustreError: 292948:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4507.105848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.106375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4507.108899] Lustre: Skipped 1 previous similar message [ 4519.392130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4519.446884] Lustre: server umount lustre-MDT0000 complete [ 4520.710280] LustreError: 292061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888556 with bad export cookie 3210482419834574679 [ 4520.713359] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4520.714455] LustreError: 292061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4520.839793] Lustre: server umount lustre-MDT0001 complete [ 4524.621312] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 4525.621632] Key type lgssc unregistered [ 4525.761277] LNet: 294054:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4525.763455] LNetError: 294054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4525.773304] LNet: Removed LNI 192.168.202.144@tcp [ 4526.054055] Key type .llcrypt unregistered [ 4526.055187] Key type ._llcrypt unregistered [ 4529.431094] Key type ._llcrypt registered [ 4529.432328] Key type .llcrypt registered [ 4529.472520] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 4529.868916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4529.901418] alg: No test for adler32 (adler32-zlib) [ 4530.777394] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 4530.857311] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 4532.432106] Key type lgssc registered [ 4532.758306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4567.228427] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 4570.000730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4570.006437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.098151] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4571.105735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4571.168911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4572.396670] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4575.177821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4575.206651] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4575.216366] Lustre: 296769:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4575.312769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4576.511357] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4578.236550] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4579.393464] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4594.594696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4594.666315] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4594.734322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4596.547129] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4598.221538] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4598.933496] Lustre: server umount lustre-OST0000 complete [ 4600.801226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4600.803370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4600.807634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4601.840204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4601.840768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4606.237207] Lustre: server umount lustre-MDT0000 complete [ 4607.511411] LustreError: 296761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888643 with bad export cookie 16955096657442208328 [ 4607.514470] LustreError: 296761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4607.514586] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4607.648330] Lustre: server umount lustre-MDT0001 complete [ 4611.063369] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 4611.945492] Key type lgssc unregistered [ 4612.065337] LNet: 299974:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4612.067732] LNetError: 299974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4612.073324] LNet: Removed LNI 192.168.202.144@tcp [ 4612.343095] Key type .llcrypt unregistered [ 4612.344214] Key type ._llcrypt unregistered [ 4621.194280] Key type ._llcrypt registered [ 4621.195353] Key type .llcrypt registered [ 4621.231750] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 4621.530763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4621.579508] alg: No test for adler32 (adler32-zlib) [ 4622.423228] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 4622.497981] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 4624.072142] Key type lgssc registered [ 4624.392463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4626.887862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4626.894309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4628.061266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.344297] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4632.127033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.262958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4633.538774] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4635.271735] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4636.542756] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4640.867611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4640.996601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4642.767805] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4644.471050] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4657.633786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4657.635693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4657.637961] Lustre: Skipped 2 previous similar messages [ 4657.640611] Lustre: Skipped 2 previous similar messages [ 4660.094447] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.44@tcp (stopping) [ 4662.755030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4662.757564] Lustre: Skipped 2 previous similar messages [ 4662.941907] Lustre: server umount lustre-MDT0000 complete [ 4664.229440] LustreError: 301331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888699 with bad export cookie 9341049892204198227 [ 4664.234414] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4664.235056] LustreError: 301331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4665.206824] LustreError: 302066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4665.212307] LustreError: 302066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4665.212884] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.44@tcp (stopping) [ 4667.873088] LustreError: 301344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4667.873458] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.920892] LustreError: 303502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4669.921158] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4669.927194] Lustre: Skipped 1 previous similar message [ 4670.374217] Lustre: server umount lustre-MDT0001 complete [ 4672.348388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4674.977718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.041240] LustreError: 300661:0:(client.c:1393:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b209dc0b x1862154225474944/t0(0) o250->MGC192.168.202.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4675.157277] LustreError: 302952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4675.165123] LustreError: 302952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4675.181969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4675.200905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4676.416405] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4679.106460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4679.239809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4680.491694] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4682.142858] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4682.724918] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4683.851522] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4684.373288] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4685.795975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4685.796164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4685.800218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4685.801913] Lustre: Skipped 1 previous similar message [ 4685.804567] Lustre: Skipped 2 previous similar messages [ 4691.741194] Lustre: server umount lustre-OST0000 complete [ 4694.496896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4694.499291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4706.273122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4706.275405] Lustre: Skipped 8 previous similar messages [ 4707.296106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4707.361711] Lustre: server umount lustre-MDT0000 complete [ 4708.630415] LustreError: 303816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888744 with bad export cookie 9341049892204205346 [ 4708.633439] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4708.635471] LustreError: 303816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4708.768733] Lustre: server umount lustre-MDT0001 complete [ 4712.301137] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 4713.313905] Key type lgssc unregistered [ 4713.433475] LNet: 307053:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4713.437805] LNetError: 307053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4713.445352] LNet: Removed LNI 192.168.202.144@tcp [ 4713.736454] Key type .llcrypt unregistered [ 4713.737638] Key type ._llcrypt unregistered [ 4722.779306] Key type ._llcrypt registered [ 4722.780388] Key type .llcrypt registered [ 4722.816377] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 4723.141099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4723.180360] alg: No test for adler32 (adler32-zlib) [ 4724.030759] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 4724.106106] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 4725.680148] Key type lgssc registered [ 4726.050870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4728.585262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4728.590727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4729.739591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4730.927512] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4733.745863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4733.880228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4735.123030] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4736.922689] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4738.090571] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4743.267083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.268218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4743.270875] Lustre: Skipped 1 previous similar message [ 4743.273513] Lustre: Skipped 1 previous similar message [ 4745.306303] Lustre: server umount lustre-MDT0000 complete [ 4746.595342] LustreError: 309817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888782 with bad export cookie 17908944922005341575 [ 4746.598111] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4746.598879] LustreError: 309817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4746.717676] Lustre: server umount lustre-MDT0001 complete [ 4751.440932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.536288] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4751.547626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4751.618787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.807262] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4755.442029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.474280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4755.485235] Lustre: 310788:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4755.581801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4756.881978] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4758.623349] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4759.826128] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4764.259426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4764.335343] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4764.339213] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4764.342494] LustreError: 312441:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 4764.345162] LustreError: 312441:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 4764.347348] LustreError: 312441:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 4764.370086] Lustre: server umount lustre-OST0000 complete [ 4764.371866] LustreError: 312441:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4769.058891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4769.128151] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4769.193036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4771.071463] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4772.768893] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4775.408463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4775.442608] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4775.471721] Lustre: lustre-OST0001: new disk, initializing [ 4775.474521] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4777.339627] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4777.364639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4777.370267] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4777.387863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4777.418038] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4779.354580] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4782.565130] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4782.565289] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4782.566329] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4782.567181] LustreError: Skipped 1 previous similar message [ 4782.570133] Lustre: Skipped 1 previous similar message [ 4786.366536] Lustre: server umount lustre-OST0001 complete [ 4787.682477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4787.683076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4787.684129] LustreError: 312908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4787.684143] LustreError: 312908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4787.686882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4787.688508] LustreError: Skipped 1 previous similar message [ 4787.692858] Lustre: Skipped 2 previous similar messages [ 4792.808627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4792.809356] LustreError: 312906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4792.813093] Lustre: Skipped 1 previous similar message [ 4792.827721] LustreError: 312906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4793.812680] Lustre: server umount lustre-OST0000 complete [ 4796.896906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4796.899482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4796.903190] Lustre: Skipped 1 previous similar message [ 4796.904817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4801.184253] Lustre: server umount lustre-MDT0000 complete [ 4802.545241] LustreError: 310823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888838 with bad export cookie 17908944922005342100 [ 4802.547915] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4802.549160] LustreError: 310823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4802.664441] Lustre: server umount lustre-MDT0001 complete [ 4806.123407] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 4807.237515] Key type lgssc unregistered [ 4807.384515] LNet: 315688:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4807.388139] LNetError: 315688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4807.394512] LNet: Removed LNI 192.168.202.144@tcp [ 4807.698375] Key type .llcrypt unregistered [ 4807.699433] Key type ._llcrypt unregistered [ 4838.132171] Key type ._llcrypt registered [ 4838.133746] Key type .llcrypt registered [ 4838.174106] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 4844.256656] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 4844.812303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4844.826867] alg: No test for adler32 (adler32-zlib) [ 4845.704268] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 4845.792564] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 4847.376134] Key type lgssc registered [ 4847.820599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4859.112886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4859.119369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4860.230618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4860.243230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4860.288149] Lustre: lustre-MDT0000: new disk, initializing [ 4860.316527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4860.323631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4861.622244] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4865.668210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4865.708242] Lustre: 320559:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4865.731655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4865.733453] Lustre: Skipped 1 previous similar message [ 4865.769644] Lustre: lustre-MDT0001: new disk, initializing [ 4865.802220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4865.814937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4865.819609] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4867.081450] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4870.215407] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4871.465824] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4873.650385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4873.748109] Lustre: lustre-OST0000: new disk, initializing [ 4873.750054] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4873.777486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4875.549901] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4877.501763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4877.507538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4877.536049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4878.619521] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4879.956668] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4880.022902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4881.290338] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4881.358444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4882.917748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4882.917785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4882.924802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4882.926442] Lustre: Skipped 1 previous similar message [ 4884.062939] Lustre: server umount lustre-OST0000 complete [ 4886.496946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4886.499920] LustreError: Skipped 1 previous similar message [ 4886.501575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4886.505036] Lustre: Skipped 1 previous similar message [ 4886.507156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4888.034218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4888.034975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.414422] Lustre: server umount lustre-MDT0000 complete [ 4892.632145] LustreError: 320595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888928 with bad export cookie 9211029459739314821 [ 4892.635547] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4892.638201] LustreError: 320595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4892.744309] Lustre: server umount lustre-MDT0001 complete [ 4895.510686] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:28:50 (1775888930) [ 4898.062630] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:28:53 (1775888933) [ 4900.883851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4901.062464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4902.357191] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4905.227633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4905.336089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4906.525769] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4908.350560] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4909.471580] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4911.606251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4913.596581] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4915.482964] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4934.456076] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4935.055471] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4935.618210] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4936.196359] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4936.847479] Lustre: Failing over lustre-MDT0000 [ 4937.022332] Lustre: server umount lustre-MDT0000 complete [ 4941.181750] LustreError: 324900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.188163] LustreError: 324900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4941.281075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4941.284271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4941.796799] LustreError: 324900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.803849] LustreError: 324900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4946.294778] LustreError: 324899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4950.214840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.258330] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4950.367688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4950.370988] Lustre: Skipped 1 previous similar message [ 4950.402496] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4951.414851] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4951.754878] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4955.625457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4955.637246] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4955.653100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4956.804197] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4957.403520] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4959.218984] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4959.857404] Lustre: Failing over lustre-MDT0000 [ 4960.038452] Lustre: server umount lustre-MDT0000 complete [ 4960.738269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4960.739323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4960.739825] LustreError: 324928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4960.739834] LustreError: 324928:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4960.751033] Lustre: Skipped 3 previous similar messages [ 4965.857794] LustreError: 324904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4965.862632] LustreError: 324904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4972.935282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4972.982874] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4973.072780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4973.100324] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4974.345511] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 4977.015286] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4978.153294] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4978.155219] Lustre: Skipped 2 previous similar messages [ 4978.162410] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4978.180452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4979.354000] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4979.929023] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4983.266627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4983.267543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4983.268374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4983.268381] Lustre: Skipped 1 previous similar message [ 4983.272244] Lustre: Skipped 3 previous similar messages [ 4988.311970] Lustre: server umount lustre-MDT0000 complete [ 4988.386121] LustreError: 324904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4988.391607] LustreError: 324904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4989.545046] LustreError: 324883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889025 with bad export cookie 9211029459739317509 [ 4989.548188] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4989.550314] LustreError: 324883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4989.666167] Lustre: server umount lustre-MDT0001 complete [ 5001.177819] Lustre: server umount lustre-OST0000 complete [ 5007.352225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5007.581295] LustreError: 330699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5007.618994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5008.970339] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5012.458077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5014.094311] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5016.325294] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5017.757288] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5020.573272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5022.847002] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5024.623481] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5032.996228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5038.054703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5038.055348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5038.061531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5038.063545] LustreError: Skipped 1 previous similar message [ 5038.068801] Lustre: Skipped 3 previous similar messages [ 5042.699678] Lustre: server umount lustre-OST0000 complete [ 5048.290376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5048.292224] Lustre: Skipped 1 previous similar message [ 5050.069126] Lustre: server umount lustre-MDT0000 complete [ 5051.680891] LustreError: 330681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889087 with bad export cookie 9211029459739318216 [ 5051.685261] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5051.689439] LustreError: 330681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5051.833548] Lustre: server umount lustre-MDT0001 complete [ 5055.517504] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 5056.686989] Key type lgssc unregistered [ 5056.810462] LNet: 333898:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5056.814452] LNetError: 333898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5056.824352] LNet: Removed LNI 192.168.202.144@tcp [ 5057.183587] Key type .llcrypt unregistered [ 5057.185392] Key type ._llcrypt unregistered [ 5067.276159] Key type ._llcrypt registered [ 5067.277161] Key type .llcrypt registered [ 5067.311291] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 5067.739400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5067.758938] alg: No test for adler32 (adler32-zlib) [ 5068.624233] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 5068.706588] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 5070.280203] Key type lgssc registered [ 5070.669199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5073.405315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5073.410855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.572057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5075.847920] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5078.691746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5078.828567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5080.131464] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5081.829677] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5083.005307] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5085.218631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5085.339975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5087.177677] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5088.813219] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5099.687374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5109.729069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5109.731228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5109.735120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5110.241594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5110.243408] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5110.247729] Lustre: Skipped 1 previous similar message [ 5114.396114] Lustre: server umount lustre-MDT0000 complete [ 5115.361119] LustreError: 335386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.365785] LustreError: 335386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5115.552229] LustreError: 335365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889151 with bad export cookie 1128733434310044169 [ 5115.555130] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5115.555298] LustreError: 335365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5115.670604] Lustre: server umount lustre-MDT0001 complete [ 5127.129976] Lustre: server umount lustre-OST0000 complete [ 5131.418567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5131.594295] LustreError: 338780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5131.620968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5132.943926] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5135.812321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5135.945689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5137.232098] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5139.008795] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5140.175925] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5142.400283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5144.290386] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5146.004806] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5156.907875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5171.680917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5171.685183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5171.692056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5171.695329] Lustre: Skipped 2 previous similar messages [ 5174.745296] Lustre: server umount lustre-MDT0000 complete [ 5175.916152] LustreError: 340393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889211 with bad export cookie 1128733434310045275 [ 5175.919502] LustreError: 340393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5175.920089] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5176.021477] Lustre: server umount lustre-MDT0001 complete [ 5187.559039] Lustre: server umount lustre-OST0000 complete [ 5192.733060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5192.903642] LustreError: 342374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5192.929145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5192.931478] Lustre: Skipped 1 previous similar message [ 5194.281303] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5197.146821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5198.566449] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5200.393273] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5201.638383] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5203.916563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5205.981579] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5207.826581] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5217.383103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5222.884080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5222.885203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5222.885538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5222.885544] Lustre: Skipped 3 previous similar messages [ 5222.887880] Lustre: Skipped 3 previous similar messages [ 5225.425518] Lustre: server umount lustre-OST0000 complete [ 5228.002322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5228.002681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5228.005173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5228.010053] Lustre: Skipped 1 previous similar message [ 5228.011826] Lustre: Skipped 1 previous similar message [ 5232.798607] Lustre: server umount lustre-MDT0000 complete [ 5233.120958] LustreError: 342374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5233.125492] LustreError: 342374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5233.994963] LustreError: 342400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889269 with bad export cookie 1128733434310046283 [ 5233.997902] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5233.998721] LustreError: 342400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5234.117205] Lustre: server umount lustre-MDT0001 complete [ 5237.572740] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 5238.505535] Key type lgssc unregistered [ 5238.636395] LNet: 345572:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5238.639312] LNetError: 345572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5238.647485] LNet: Removed LNI 192.168.202.144@tcp [ 5238.958625] Key type .llcrypt unregistered [ 5238.959710] Key type ._llcrypt unregistered [ 5248.295483] Key type ._llcrypt registered [ 5248.297628] Key type .llcrypt registered [ 5248.337324] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 5248.694203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5248.712544] alg: No test for adler32 (adler32-zlib) [ 5249.571123] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 5249.668192] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 5251.256137] Key type lgssc registered [ 5251.669815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5254.193389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5254.199639] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5254.201981] LustreError: 346986:0:(osd_handler.c:8532:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5254.210032] LustreError: 346986:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5254.211962] LustreError: 346986:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5254.213886] LustreError: 346986:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5254.216045] LustreError: 346986:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5256.510455] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5256.514086] LustreError: 347382:0:(osd_handler.c:8532:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5256.523883] LustreError: 347382:0:(obd_config.c:843:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5256.526255] LustreError: 347382:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5256.528583] LustreError: 347382:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5256.530894] LustreError: 347382:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5262.842844] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 5263.760352] Key type lgssc unregistered [ 5263.885348] LNet: 348366:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5263.887595] LNetError: 348366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5263.897272] LNet: Removed LNI 192.168.202.144@tcp [ 5264.153421] Key type .llcrypt unregistered [ 5264.154881] Key type ._llcrypt unregistered [ 5269.961284] Key type ._llcrypt registered [ 5269.962261] Key type .llcrypt registered [ 5269.995151] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 5276.137743] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 5276.511498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5276.543764] alg: No test for adler32 (adler32-zlib) [ 5277.386794] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 5277.465948] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 5279.040243] Key type lgssc registered [ 5279.508624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5291.268806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5291.276660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5292.378085] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5292.389718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5292.433619] Lustre: lustre-MDT0000: new disk, initializing [ 5292.456251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5292.462135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5293.753744] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5297.840915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5297.879242] Lustre: 351462:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5297.903297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5297.905919] Lustre: Skipped 1 previous similar message [ 5297.945207] Lustre: lustre-MDT0001: new disk, initializing [ 5297.977832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5297.989115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5297.993563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5299.277447] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5302.470643] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5303.588301] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5306.344648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5306.488290] Lustre: lustre-OST0000: new disk, initializing [ 5306.490619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5306.524779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5308.420924] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5309.751904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5309.757594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5309.785148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5311.451764] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5312.768456] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5312.855510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5314.186780] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5314.280853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5315.043632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5315.046457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5315.047574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5315.050308] Lustre: Skipped 1 previous similar message [ 5320.166402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5320.169068] Lustre: Skipped 2 previous similar messages [ 5325.287812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5325.290945] Lustre: Skipped 1 previous similar message [ 5329.376118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5329.457405] Lustre: server umount lustre-OST0000 complete [ 5333.985362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5333.989102] LustreError: Skipped 1 previous similar message [ 5333.990685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5333.996889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5335.521955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5337.325276] Lustre: server umount lustre-MDT0000 complete [ 5338.545504] LustreError: 351453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889374 with bad export cookie 2131752677503720929 [ 5338.548532] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5338.551236] LustreError: 351453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5338.672258] Lustre: server umount lustre-MDT0001 complete [ 5341.339259] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5341.862057] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:36:17 (1775889377) [ 5343.975786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5344.160922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5345.448298] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5348.495272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.624618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5349.901247] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5351.656913] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5352.805871] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5354.988164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5356.868434] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5359.000505] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5361.378458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5361.436748] Lustre: lustre-OST0001: new disk, initializing [ 5361.438549] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5361.465838] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5361.468871] Lustre: Skipped 1 previous similar message [ 5363.310502] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5364.121560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5364.127838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5364.165413] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5366.531279] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5369.319641] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5369.319991] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5369.321635] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5369.321641] Lustre: Skipped 2 previous similar messages [ 5369.325145] Lustre: Skipped 1 previous similar message [ 5369.330152] LustreError: Skipped 1 previous similar message [ 5374.489767] Lustre: server umount lustre-OST0001 complete [ 5375.463181] LustreError: 357110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5375.469317] LustreError: 357110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5380.586451] LustreError: 357111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5380.594784] LustreError: 357111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5385.704126] LustreError: 357111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5385.708954] LustreError: 357111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5390.305150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5390.307591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5390.312594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5390.315568] Lustre: Skipped 4 previous similar messages [ 5390.821825] LustreError: 357111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5390.828523] LustreError: 357111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5392.980241] Lustre: server umount lustre-OST0000 complete [ 5394.913096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5394.915234] LustreError: Skipped 1 previous similar message [ 5400.408967] Lustre: server umount lustre-MDT0000 complete [ 5401.056930] LustreError: 355517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5401.060993] LustreError: 355517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5401.595354] LustreError: 355496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889437 with bad export cookie 2131752677503721797 [ 5401.598343] LustreError: 355496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5401.601093] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5407.796349] Lustre: server umount lustre-MDT0001 complete [ 5411.218942] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 5412.202382] Key type lgssc unregistered [ 5412.335596] LNet: 359866:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5412.338712] LNetError: 359866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5412.350526] LNet: Removed LNI 192.168.202.144@tcp [ 5412.653279] Key type .llcrypt unregistered [ 5412.654331] Key type ._llcrypt unregistered [ 5430.618504] Key type ._llcrypt registered [ 5430.619584] Key type .llcrypt registered [ 5430.656305] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 5431.081958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5431.091494] alg: No test for adler32 (adler32-zlib) [ 5431.934975] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 5432.012542] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 5433.592101] Key type lgssc registered [ 5433.918758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5436.447043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5436.452823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5437.545772] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5437.556464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5437.622620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5438.899280] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5441.853562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5441.889452] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5441.901747] Lustre: 362622:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5442.010309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5443.324147] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5445.149089] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5446.323774] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5447.136972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5447.139273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5447.143356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5449.185746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5449.186378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5453.079756] Lustre: server umount lustre-MDT0000 complete [ 5454.303179] LustreError: 363346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889490 with bad export cookie 9893547340178228695 [ 5454.306613] LustreError: 363346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5454.306638] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5454.433431] Lustre: server umount lustre-MDT0001 complete [ 5457.406881] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:38:12 (1775889492) [ 5459.598487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5459.803559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5461.137477] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5463.849333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5463.967856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5465.298233] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5466.956180] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5468.057731] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5470.143656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5470.226098] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5470.294298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5472.160354] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5473.866410] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5477.480061] Lustre: Permanently deactivating lustre-OST0000 [ 5477.484852] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5479.728766] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5480.663764] LustreError: 367451:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5480.666917] LustreError: 367451:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5480.668976] LustreError: 367451:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5481.272072] LustreError: 367499:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5481.274174] LustreError: 367499:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5488.017336] Lustre: server umount lustre-OST0000 complete [ 5488.922421] LustreError: 367748:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5488.925960] LustreError: 367748:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5489.632942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5489.635123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5489.639774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5489.642341] Lustre: Skipped 1 previous similar message [ 5491.681258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5491.681914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5496.801168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5496.802695] Lustre: Skipped 2 previous similar messages [ 5499.910389] Lustre: server umount lustre-MDT0000 complete [ 5501.317141] LustreError: 364887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889537 with bad export cookie 9893547340178229262 [ 5501.323883] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5511.717370] Lustre: server umount lustre-MDT0001 complete [ 5514.565122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.804297] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5516.715494] LustreError: 368888:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5518.833473] Lustre: 369227:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.144@tcp [ 5519.576890] Lustre: server umount MGS complete [ 5523.333430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5523.478625] LustreError: 369951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5523.483053] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5523.483919] LustreError: 369951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5523.485465] Lustre: Skipped 1 previous similar message [ 5523.506311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5524.706118] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5527.447726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.552536] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5528.762875] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5530.492099] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5531.651439] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5532.289340] Lustre: Permanently reactivating lustre-OST0000 [ 5532.294541] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5532.296337] Lustre: Skipped 2 previous similar messages [ 5554.656100] LustreError: 371406:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5557.084671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5557.244885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5557.247883] Lustre: Skipped 1 previous similar message [ 5559.295866] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5561.255176] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5574.629416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5574.629439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5574.634740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5574.637939] Lustre: Skipped 1 previous similar message [ 5574.642032] LustreError: Skipped 1 previous similar message [ 5575.608138] Lustre: server umount lustre-OST0000 complete [ 5578.721791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5579.538306] Lustre: server umount lustre-MDT0000 complete [ 5579.746155] LustreError: 372082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5579.751256] LustreError: 372082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5581.106833] LustreError: 369933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889616 with bad export cookie 9893547340178230298 [ 5581.110612] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5581.114783] LustreError: 369933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5581.273636] Lustre: server umount lustre-MDT0001 complete [ 5585.398663] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 5586.555345] Key type lgssc unregistered [ 5586.684470] LNet: 374121:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5586.687252] LNetError: 374121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5586.695494] LNet: Removed LNI 192.168.202.144@tcp [ 5587.049990] Key type .llcrypt unregistered [ 5587.051052] Key type ._llcrypt unregistered [ 5594.527963] Key type ._llcrypt registered [ 5594.529501] Key type .llcrypt registered [ 5594.583536] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 5601.661711] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 5602.208923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5602.216924] alg: No test for adler32 (adler32-zlib) [ 5603.067741] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 5603.149769] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 5604.728168] Key type lgssc registered [ 5605.194389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5620.646331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5620.652865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5621.766588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5621.779751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5621.824705] Lustre: lustre-MDT0000: new disk, initializing [ 5621.851385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5621.857373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5623.244939] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5628.142282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5628.181289] Lustre: 377218:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5628.210994] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5628.214062] Lustre: Skipped 1 previous similar message [ 5628.260778] Lustre: lustre-MDT0001: new disk, initializing [ 5628.312154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5628.332213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5628.338186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5630.048494] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5633.519641] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5634.788952] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5637.215147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5637.312185] Lustre: lustre-OST0000: new disk, initializing [ 5637.313886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5637.339778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5639.490554] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5642.728448] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5644.619327] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5645.827652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5645.830842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5645.859897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5646.765156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5647.976580] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5648.054076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5650.915843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5650.916650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5650.918473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5650.923798] Lustre: Skipped 1 previous similar message [ 5650.928562] Lustre: Skipped 1 previous similar message [ 5655.081056] Lustre: server umount lustre-OST0000 complete [ 5659.105363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5659.109266] LustreError: Skipped 1 previous similar message [ 5659.111516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5659.117447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5661.153244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5661.153695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5662.564080] Lustre: server umount lustre-MDT0000 complete [ 5663.763566] LustreError: 377209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889699 with bad export cookie 7391851703697715715 [ 5663.768505] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5663.887354] Lustre: server umount lustre-MDT0001 complete [ 5666.473645] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5667.008438] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:41:42 (1775889702) [ 5669.316536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5669.508714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5670.826852] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5673.427015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.555343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5674.788378] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5676.437462] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5677.527039] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5679.713871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5681.771089] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5683.927572] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5684.789445] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5695.969365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5699.552886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5699.555310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.559261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5699.560923] Lustre: Skipped 1 previous similar message [ 5703.627501] Lustre: server umount lustre-OST0000 complete [ 5705.698198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.699141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5705.706796] Lustre: Skipped 2 previous similar messages [ 5705.709672] Lustre: Skipped 2 previous similar messages [ 5711.215154] Lustre: server umount lustre-MDT0000 complete [ 5712.568904] LustreError: 381260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889748 with bad export cookie 7391851703697716569 [ 5712.573816] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5712.576366] LustreError: 381260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5712.723785] Lustre: server umount lustre-MDT0001 complete [ 5717.002310] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 5718.572702] Key type lgssc unregistered [ 5718.738690] LNet: 384669:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5718.740833] LNetError: 384669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5718.747553] LNet: Removed LNI 192.168.202.144@tcp [ 5719.034767] Key type .llcrypt unregistered [ 5719.036032] Key type ._llcrypt unregistered [ 5727.413581] Key type ._llcrypt registered [ 5727.414762] Key type .llcrypt registered [ 5727.463179] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 5727.612369] Key type .llcrypt unregistered [ 5727.613640] Key type ._llcrypt unregistered [ 5734.531863] Key type ._llcrypt registered [ 5734.532976] Key type .llcrypt registered [ 5734.575630] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 5734.963553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5735.015334] alg: No test for adler32 (adler32-zlib) [ 5735.894694] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 5735.977198] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 5737.552189] Key type lgssc registered [ 5737.989890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5740.564269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5740.575063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5741.774465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5743.077388] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5746.414500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5746.572125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5748.711804] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5750.886316] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5751.717986] LustreError: 387547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5751.725081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5751.728425] LustreError: 387547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5753.639811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5753.788959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5755.025899] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5764.068523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5764.070912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5764.077336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5764.084751] Lustre: Skipped 1 previous similar message [ 5769.126393] Lustre: server umount lustre-OST0000 complete [ 5772.771546] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5772.776154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5772.783273] Lustre: Skipped 1 previous similar message [ 5772.786288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5774.306104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5774.307298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.425961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5779.429913] Lustre: Skipped 2 previous similar messages [ 5783.522643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5783.527100] Lustre: Skipped 1 previous similar message [ 5785.056163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5785.125790] Lustre: server umount lustre-MDT0000 complete [ 5786.831997] LustreError: 388437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889822 with bad export cookie 16698974094531697017 [ 5786.839277] LustreError: 388437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5786.839468] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5786.990366] Lustre: server umount lustre-MDT0001 complete [ 5791.035454] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 5792.509074] Key type lgssc unregistered [ 5792.654640] LNet: 389810:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5792.659533] LNetError: 389810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5792.668649] LNet: Removed LNI 192.168.202.144@tcp [ 5793.113415] Key type .llcrypt unregistered [ 5793.115305] Key type ._llcrypt unregistered [ 5804.261002] Key type ._llcrypt registered [ 5804.263133] Key type .llcrypt registered [ 5804.332668] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 5804.768472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5804.774745] alg: No test for adler32 (adler32-zlib) [ 5805.619111] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 5805.695690] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 5807.288191] Key type lgssc registered [ 5807.837752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5811.052056] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5811.058303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5826.528442] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5831.648297] LustreError: 391160:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.144@tcp: failed processing log, type 4: rc = -110 [ 5858.272204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.406739] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5862.976365] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5866.208600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5866.409502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5866.451709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5867.732774] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5871.588491] LustreError: 391186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.598365] LustreError: 391186:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5871.768671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5871.931670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5873.607234] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5876.097521] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5877.835355] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5881.825536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5881.830637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5881.839252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5882.338308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5882.343188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5882.351948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5883.366113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5883.371415] Lustre: Skipped 1 previous similar message [ 5886.760959] Lustre: server umount lustre-OST0000 complete [ 5891.044075] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5891.049107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.057893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5893.602679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.707799] Lustre: server umount lustre-MDT0000 complete [ 5896.479999] LustreError: 391167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889932 with bad export cookie 5395032056350027224 [ 5896.486422] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5896.487052] LustreError: 391167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5896.627986] Lustre: server umount lustre-MDT0001 complete [ 5901.035780] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 5902.553088] Key type lgssc unregistered [ 5902.703365] LNet: 394375:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5902.706972] LNetError: 394375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5902.715284] LNet: Removed LNI 192.168.202.144@tcp [ 5903.109860] Key type .llcrypt unregistered [ 5903.112075] Key type ._llcrypt unregistered [ 5913.924919] Key type ._llcrypt registered [ 5913.926197] Key type .llcrypt registered [ 5913.965187] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 5914.390884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5914.397122] alg: No test for adler32 (adler32-zlib) [ 5915.252250] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 5915.330654] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 5916.920148] Key type lgssc registered [ 5917.469885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5921.140043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5921.150848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5922.374740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5923.915932] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5927.620786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5927.801444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5929.433726] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5931.924463] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5933.567096] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5936.752671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5936.914772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5939.277583] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 5941.770000] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5948.205071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5957.509573] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.44@tcp (stopping) [ 5958.630765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5958.631431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5958.633826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5958.639878] Lustre: Skipped 2 previous similar messages [ 5959.992691] Lustre: server umount lustre-MDT0000 complete [ 5960.913991] LustreError: 395732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.44@tcp arrived at 1775889996 with bad export cookie 5923659161528812894 [ 5961.856867] LustreError: 396469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775889997 with bad export cookie 5923659161528812271 [ 5961.864057] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5961.864288] LustreError: 396469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5963.745790] LustreError: 396462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5963.746833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5963.747759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5963.748768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5963.748775] Lustre: Skipped 2 previous similar messages [ 5963.755420] LustreError: 396462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5966.753662] LustreError: 397906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5966.754444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5966.767057] LustreError: 397906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5967.984208] Lustre: server umount lustre-OST0000 complete [ 5971.115504] Lustre: server umount lustre-MDT0001 complete [ 5975.543610] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 5976.463697] Key type lgssc unregistered [ 5976.579496] LNet: 399102:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5976.581649] LNetError: 399102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5976.594328] LNet: Removed LNI 192.168.202.144@tcp [ 5976.965094] Key type .llcrypt unregistered [ 5976.967280] Key type ._llcrypt unregistered [ 5987.589254] Key type ._llcrypt registered [ 5987.591198] Key type .llcrypt registered [ 5987.660784] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 5988.098103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5988.112446] alg: No test for adler32 (adler32-zlib) [ 5988.962083] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 5989.058819] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 5990.648117] Key type lgssc registered [ 5991.076280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5994.736945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5994.749215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5995.940833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5997.610648] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6001.377507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6001.495541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6003.429214] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6005.967201] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6007.314811] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6010.703480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6010.893719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6013.116534] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6015.510208] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6021.158761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6031.235064] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.44@tcp (stopping) [ 6031.330433] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6031.330954] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6033.324309] Lustre: server umount lustre-MDT0001 complete [ 6036.449967] LustreError: 400493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6036.450282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6036.454971] LustreError: 400493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6036.458605] Lustre: Skipped 1 previous similar message [ 6036.462640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6036.462850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6036.466835] Lustre: Skipped 2 previous similar messages [ 6041.569167] LustreError: 400494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6041.570156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6041.767575] Lustre: server umount lustre-OST0000 complete [ 6043.666287] Lustre: server umount lustre-MDT0000 complete [ 6049.806896] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 6051.239984] Key type lgssc unregistered [ 6051.359517] LNet: 403845:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6051.361842] LNetError: 403845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6051.371559] LNet: Removed LNI 192.168.202.144@tcp [ 6051.682750] Key type .llcrypt unregistered [ 6051.684215] Key type ._llcrypt unregistered [ 6062.254642] Key type ._llcrypt registered [ 6062.255902] Key type .llcrypt registered [ 6062.292860] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 6069.858690] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 6070.308048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6070.321109] alg: No test for adler32 (adler32-zlib) [ 6071.180592] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 6071.257173] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 6072.840149] Key type lgssc registered [ 6073.426895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6090.479791] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 6098.249870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6098.260695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6099.378072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6099.394519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6099.436847] Lustre: lustre-MDT0000: new disk, initializing [ 6099.466968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6099.475616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6101.102754] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6109.068632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6109.220804] Lustre: lustre-OST0000: new disk, initializing [ 6109.223480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6109.226434] Lustre: Skipped 1 previous similar message [ 6109.259707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6111.701728] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6115.692832] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6116.885176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6116.891709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6116.906506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6128.473378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6128.546658] Lustre: 407781:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6128.603474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6128.650909] Lustre: lustre-MDT0001: new disk, initializing [ 6128.701123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6128.718835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6128.723431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6130.428699] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6134.869656] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6134.958767] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6136.869996] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6136.992951] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6139.875232] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6139.878211] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6139.881179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6139.887620] Lustre: Skipped 1 previous similar message [ 6143.877574] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.44@tcp (stopping) [ 6143.882146] Lustre: Skipped 1 previous similar message [ 6144.996884] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6145.000785] Lustre: Skipped 1 previous similar message [ 6145.393130] Lustre: server umount lustre-MDT0001 complete [ 6150.116247] LustreError: 407788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6150.126026] LustreError: 407788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6155.237107] LustreError: 407789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6155.249060] LustreError: 407789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6160.355864] LustreError: 407793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6160.357772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.374412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6164.335139] Lustre: server umount lustre-OST0000 complete [ 6165.477071] LustreError: 407794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6172.474838] Lustre: server umount lustre-MDT0000 complete [ 6178.608707] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 6180.103219] Key type lgssc unregistered [ 6180.270634] LNet: 412354:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6180.275525] LNetError: 412354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6180.291552] LNet: Removed LNI 192.168.202.144@tcp [ 6180.731784] Key type .llcrypt unregistered [ 6180.733885] Key type ._llcrypt unregistered [ 6198.723836] Key type ._llcrypt registered [ 6198.724813] Key type .llcrypt registered [ 6198.755486] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 6198.836343] Key type .llcrypt unregistered [ 6198.837481] Key type ._llcrypt unregistered [ 6205.317131] Key type ._llcrypt registered [ 6205.318111] Key type .llcrypt registered [ 6205.352183] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 6205.752061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6205.791675] alg: No test for adler32 (adler32-zlib) [ 6206.642712] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 6206.730934] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 6208.320094] Key type lgssc registered [ 6208.732283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6217.668535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6217.673188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6218.789326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6218.805792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6218.848563] Lustre: lustre-MDT0000: new disk, initializing [ 6218.889221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6218.900731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6220.577389] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6225.758114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6225.822215] Lustre: 415669:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6225.861136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6225.864611] Lustre: Skipped 1 previous similar message [ 6225.913073] Lustre: lustre-MDT0001: new disk, initializing [ 6225.949989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6225.964766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6225.968149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6227.582816] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6231.447898] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6232.939860] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6235.891244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6236.055496] Lustre: lustre-OST0000: new disk, initializing [ 6236.058196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6236.102375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6238.826887] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6242.838634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6242.847563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6242.907338] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6242.908660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6246.368766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6246.371029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.374432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6247.912074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6247.912386] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6252.030223] Lustre: server umount lustre-MDT0000 complete [ 6253.847531] LustreError: 415660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775890289 with bad export cookie 2087367769381134186 [ 6253.854992] LustreError: 415660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6253.855466] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6254.024340] Lustre: server umount lustre-MDT0001 complete [ 6266.172584] Lustre: server umount lustre-OST0000 complete [ 6271.046646] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 02:51:46 (1775890306) [ 6274.824387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6275.049985] LustreError: 419323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6275.059568] LustreError: 419323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6275.094125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.648614] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6280.162099] LustreError: 419324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6280.886893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6281.090980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6282.570567] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6285.328422] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6286.773598] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6303.141665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6303.240660] Lustre: Found index 0 for lustre-OST0000, updating log [ 6303.320094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6305.765824] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6307.893083] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6319.717811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6325.220866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6325.222464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6325.223617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6325.223626] Lustre: Skipped 2 previous similar messages [ 6325.228845] Lustre: Skipped 2 previous similar messages [ 6331.308542] Lustre: server umount lustre-OST0000 complete [ 6344.151206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6344.151983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6344.158387] Lustre: Skipped 1 previous similar message [ 6344.163506] Lustre: Skipped 4 previous similar messages [ 6349.282218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6349.287688] Lustre: Skipped 1 previous similar message [ 6350.321471] Lustre: server umount lustre-MDT0000 complete [ 6352.061770] LustreError: 419303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775890387 with bad export cookie 2087367769381135565 [ 6352.067910] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6352.069102] LustreError: 419303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6352.222465] Lustre: server umount lustre-MDT0001 complete [ 6356.106164] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 02:53:11 (1775890391) [ 6367.520286] LNet: 1 local NIs in recovery (showing 1): 192.168.202.144@tcp [ 6942.384151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6942.549155] LustreError: 425422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6942.570234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6943.804082] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6947.293299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6947.469944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6949.155367] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6951.059084] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6952.835179] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6968.550486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6968.664612] Lustre: Found index 0 for lustre-OST0000, updating log [ 6968.772084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6970.758826] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 6970.855544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6972.931320] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6974.886548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.895931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6975.906644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6975.908290] Lustre: Skipped 1 previous similar message [ 6980.005934] Lustre: server umount lustre-OST0000 complete [ 6982.179421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.181485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6982.187267] Lustre: Skipped 1 previous similar message [ 6982.191178] Lustre: Skipped 1 previous similar message [ 6986.273555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6986.275971] Lustre: Skipped 8 previous similar messages [ 6987.299937] LustreError: 426134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6987.308102] LustreError: 426134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6987.325677] Lustre: server umount lustre-MDT0000 complete [ 6988.322114] LustreError: 427013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6988.330195] LustreError: 427013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6988.993405] LustreError: 427099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775891024 with bad export cookie 2087367769381136636 [ 6988.998584] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6988.999350] LustreError: 427099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6989.133247] Lustre: server umount lustre-MDT0001 complete [ 6995.232306] LNet: 1 local NIs in recovery (showing 1): 192.168.202.144@tcp [ 7007.505473] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 7011.716483] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 7026.618539] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:04:21 (1775891061) [ 7033.076177] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 7034.278630] Key type lgssc unregistered [ 7034.426521] LNet: 431692:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7034.430712] LNetError: 431692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7100.006732] LNet: Removed LNI 192.168.202.144@tcp [ 7100.471150] Key type .llcrypt unregistered [ 7100.473672] Key type ._llcrypt unregistered [ 7108.333645] Key type ._llcrypt registered [ 7108.335390] Key type .llcrypt registered [ 7108.389788] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 7108.728185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7108.776799] alg: No test for adler32 (adler32-zlib) [ 7109.634691] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 7109.729294] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 7111.320181] Key type lgssc registered [ 7111.842684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7113.219602] LNet: Added LNI 192.168.202.144@tcp100 [8/256/0/180] [ 7113.621134] LNet: Added LNI 192.168.202.144@tcp101 [8/256/0/180] [ 7114.036817] LNet: Added LNI 192.168.202.144@tcp102 [8/256/0/180] [ 7114.444707] LNet: Added LNI 192.168.202.144@tcp103 [8/256/0/180] [ 7114.724384] LNet: Added LNI 192.168.202.144@tcp104 [8/256/0/180] [ 7115.023537] LNet: Added LNI 192.168.202.144@tcp105 [8/256/0/180] [ 7115.417355] LNet: Added LNI 192.168.202.144@tcp106 [8/256/0/180] [ 7115.829831] LNet: Added LNI 192.168.202.144@tcp107 [8/256/0/180] [ 7116.257156] LNet: Added LNI 192.168.202.144@tcp108 [8/256/0/180] [ 7116.695284] LNet: Added LNI 192.168.202.144@tcp109 [8/256/0/180] [ 7117.140839] LNet: Added LNI 192.168.202.144@tcp110 [8/256/0/180] [ 7117.478241] LNet: Added LNI 192.168.202.144@tcp111 [8/256/0/180] [ 7117.744444] LNet: Added LNI 192.168.202.144@tcp112 [8/256/0/180] [ 7118.007203] LNet: Added LNI 192.168.202.144@tcp113 [8/256/0/180] [ 7118.326276] LNet: Added LNI 192.168.202.144@tcp114 [8/256/0/180] [ 7118.743863] LNet: Added LNI 192.168.202.144@tcp115 [8/256/0/180] [ 7119.154049] LNet: Added LNI 192.168.202.144@tcp116 [8/256/0/180] [ 7119.595371] LNet: Added LNI 192.168.202.144@tcp117 [8/256/0/180] [ 7120.029521] LNet: Added LNI 192.168.202.144@tcp118 [8/256/0/180] [ 7120.464746] LNet: Added LNI 192.168.202.144@tcp119 [8/256/0/180] [ 7120.899291] LNet: Added LNI 192.168.202.144@tcp120 [8/256/0/180] [ 7121.333948] LNet: Added LNI 192.168.202.144@tcp121 [8/256/0/180] [ 7121.767356] LNet: Added LNI 192.168.202.144@tcp122 [8/256/0/180] [ 7122.171137] LNet: Added LNI 192.168.202.144@tcp123 [8/256/0/180] [ 7122.542659] LNet: Added LNI 192.168.202.144@tcp124 [8/256/0/180] [ 7122.803083] LNet: Added LNI 192.168.202.144@tcp125 [8/256/0/180] [ 7123.133857] LNet: Added LNI 192.168.202.144@tcp126 [8/256/0/180] [ 7123.510668] LNet: Added LNI 192.168.202.144@tcp127 [8/256/0/180] [ 7123.892956] LNet: Added LNI 192.168.202.144@tcp128 [8/256/0/180] [ 7124.288961] LNet: Added LNI 192.168.202.144@tcp129 [8/256/0/180] [ 7124.708325] LNet: Added LNI 192.168.202.144@tcp130 [8/256/0/180] [ 7125.150548] LNet: Added LNI 192.168.202.144@tcp131 [8/256/0/180] [ 7125.583444] LNet: Added LNI 192.168.202.144@tcp132 [8/256/0/180] [ 7126.010833] LNet: Added LNI 192.168.202.144@tcp133 [8/256/0/180] [ 7126.428593] LNet: Added LNI 192.168.202.144@tcp134 [8/256/0/180] [ 7126.863819] LNet: Added LNI 192.168.202.144@tcp135 [8/256/0/180] [ 7129.435304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7129.443840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.494675] Lustre: 434671:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.144@tcp: can't add NID 192.168.202.144@tcp130: rc = -75 [ 7130.551647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7130.563598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7130.599137] Lustre: lustre-MDT0000: new disk, initializing [ 7130.630810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7130.637554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7132.257457] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7137.584037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.651533] Lustre: 434688:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7137.715652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7137.716470] Lustre: 435471:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.144@tcp: can't add NID 192.168.202.144@tcp130: rc = -75 [ 7137.717608] Lustre: Skipped 1 previous similar message [ 7137.720584] Lustre: 435471:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7137.731035] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7137.757612] Lustre: lustre-MDT0001: new disk, initializing [ 7137.800574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7137.812089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7137.814881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7138.962130] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7142.770248] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7143.870506] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7147.181799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7147.357872] Lustre: lustre-OST0000: new disk, initializing [ 7147.361682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7147.420515] Lustre: 436438:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.144@tcp: can't add NID 192.168.202.144@tcp130: rc = -75 [ 7147.427814] Lustre: 436438:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 7147.458751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7149.186593] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7151.229214] Lustre: 436901:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.144@tcp: can't add NID 192.168.202.144@tcp130: rc = -75 [ 7151.234753] Lustre: 436901:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7151.263698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7151.270333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7151.335683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7153.002564] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7156.708271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7156.710857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7156.713147] LustreError: Skipped 1 previous similar message [ 7156.722409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7156.727089] Lustre: Skipped 1 previous similar message [ 7157.672633] Lustre: server umount lustre-OST0000 complete [ 7161.827215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7161.828873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7161.835199] Lustre: Skipped 2 previous similar messages [ 7161.843646] Lustre: Skipped 1 previous similar message [ 7165.054637] Lustre: server umount lustre-MDT0000 complete [ 7166.802170] LustreError: 435468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775891202 with bad export cookie 14284198674830971442 [ 7166.809077] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7166.809337] LustreError: 435468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7166.954140] Lustre: server umount lustre-MDT0001 complete [ 7170.775830] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 7172.068232] Key type lgssc unregistered [ 7172.260618] LNet: 438215:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7172.266893] LNetError: 438215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7172.272099] LNet: Removed LNI 192.168.202.144@tcp [ 7173.280229] LNet: Removed LNI 192.168.202.144@tcp100 [ 7173.283343] LNetError: 438215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7173.288745] LNetError: 438215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7173.294392] LNet: Removed LNI 192.168.202.144@tcp101 [ 7174.304150] LNet: Removed LNI 192.168.202.144@tcp102 [ 7174.307325] LNetError: 438215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7174.314419] LNetError: 438215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7174.320163] LNet: Removed LNI 192.168.202.144@tcp103 [ 7174.323874] LNet: Removed LNI 192.168.202.144@tcp104 [ 7174.328240] LNet: Removed LNI 192.168.202.144@tcp105 [ 7174.332318] LNet: Removed LNI 192.168.202.144@tcp106 [ 7174.336808] LNet: Removed LNI 192.168.202.144@tcp107 [ 7174.340674] LNet: Removed LNI 192.168.202.144@tcp108 [ 7174.343410] LNet: Removed LNI 192.168.202.144@tcp109 [ 7174.346256] LNet: Removed LNI 192.168.202.144@tcp110 [ 7174.349091] LNet: Removed LNI 192.168.202.144@tcp111 [ 7174.351870] LNet: Removed LNI 192.168.202.144@tcp112 [ 7174.354572] LNet: Removed LNI 192.168.202.144@tcp113 [ 7174.357807] LNet: Removed LNI 192.168.202.144@tcp114 [ 7174.360828] LNet: Removed LNI 192.168.202.144@tcp115 [ 7174.363318] LNet: Removed LNI 192.168.202.144@tcp116 [ 7174.365859] LNet: Removed LNI 192.168.202.144@tcp117 [ 7174.368417] LNet: Removed LNI 192.168.202.144@tcp118 [ 7174.370940] LNet: Removed LNI 192.168.202.144@tcp119 [ 7174.373413] LNet: Removed LNI 192.168.202.144@tcp120 [ 7174.375914] LNet: Removed LNI 192.168.202.144@tcp121 [ 7174.378340] LNet: Removed LNI 192.168.202.144@tcp122 [ 7174.380756] LNet: Removed LNI 192.168.202.144@tcp123 [ 7174.383170] LNet: Removed LNI 192.168.202.144@tcp124 [ 7174.385489] LNet: Removed LNI 192.168.202.144@tcp125 [ 7174.387761] LNet: Removed LNI 192.168.202.144@tcp126 [ 7174.389993] LNet: Removed LNI 192.168.202.144@tcp127 [ 7174.392348] LNet: Removed LNI 192.168.202.144@tcp128 [ 7174.394496] LNet: Removed LNI 192.168.202.144@tcp129 [ 7174.396537] LNet: Removed LNI 192.168.202.144@tcp130 [ 7174.398618] LNet: Removed LNI 192.168.202.144@tcp131 [ 7174.400599] LNet: Removed LNI 192.168.202.144@tcp132 [ 7174.402606] LNet: Removed LNI 192.168.202.144@tcp133 [ 7174.404500] LNet: Removed LNI 192.168.202.144@tcp134 [ 7174.412552] LNet: Removed LNI 192.168.202.144@tcp135 [ 7174.737238] Key type .llcrypt unregistered [ 7174.739346] Key type ._llcrypt unregistered [ 7184.117752] Key type ._llcrypt registered [ 7184.118746] Key type .llcrypt registered [ 7184.150533] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 7190.605812] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 7190.982438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7191.008456] alg: No test for adler32 (adler32-zlib) [ 7191.886402] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 7191.979614] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 7193.568197] Key type lgssc registered [ 7193.992365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7209.016775] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:07:24 (1775891244) [ 7216.482971] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 7217.931809] Key type lgssc unregistered [ 7218.094670] LNet: 442221:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7218.099423] LNetError: 442221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7218.112620] LNet: Removed LNI 192.168.202.144@tcp [ 7218.542601] Key type .llcrypt unregistered [ 7218.544924] Key type ._llcrypt unregistered [ 7226.534598] Key type ._llcrypt registered [ 7226.536066] Key type .llcrypt registered [ 7226.582832] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 7226.893390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7226.899784] alg: No test for adler32 (adler32-zlib) [ 7227.769792] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 7227.863945] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 7229.440191] Key type lgssc registered [ 7229.931964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7233.457130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7233.466331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7234.563447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7234.574386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7234.615067] Lustre: lustre-MDT0000: new disk, initializing [ 7234.641614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7234.650505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7236.195538] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7241.898805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7241.973618] Lustre: 443394:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7242.020299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7242.023745] Lustre: Skipped 1 previous similar message [ 7242.076250] Lustre: lustre-MDT0001: new disk, initializing [ 7242.131323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7242.151631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7242.156765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7243.992467] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7247.991431] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7249.561519] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7252.383097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7252.549865] Lustre: lustre-OST0000: new disk, initializing [ 7252.553884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7252.602454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7254.956598] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7258.692603] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7260.057888] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing lnet_if_list [ 7261.201621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7261.207922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7261.262276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7262.156250] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7262.175310] LNet: Added LNI 192.168.202.144@tcp10 [8/256/0/180] [ 7264.346249] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7264.364342] LNet: Added LNI 192.168.202.144@tcp11 [8/256/0/180] [ 7265.969708] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7265.988804] LNet: Added LNI 192.168.202.144@tcp12 [8/256/0/180] [ 7267.520909] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7267.533371] LNet: Added LNI 192.168.202.144@tcp13 [8/256/0/180] [ 7269.380240] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7269.399206] LNet: Added LNI 192.168.202.144@tcp14 [8/256/0/180] [ 7270.951850] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7270.968487] LNet: Added LNI 192.168.202.144@tcp15 [8/256/0/180] [ 7272.769792] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7272.788677] LNet: Added LNI 192.168.202.144@tcp16 [8/256/0/180] [ 7274.683464] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7274.703110] LNet: Added LNI 192.168.202.144@tcp17 [8/256/0/180] [ 7276.248940] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7276.267967] LNet: Added LNI 192.168.202.144@tcp18 [8/256/0/180] [ 7278.359414] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7278.371544] LNet: Added LNI 192.168.202.144@tcp19 [8/256/0/180] [ 7280.139309] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7280.147678] LNet: Added LNI 192.168.202.144@tcp20 [8/256/0/180] [ 7281.813612] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7281.825875] LNet: Added LNI 192.168.202.144@tcp21 [8/256/0/180] [ 7283.971516] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7283.986116] LNet: Added LNI 192.168.202.144@tcp22 [8/256/0/180] [ 7285.795736] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7285.804307] LNet: Added LNI 192.168.202.144@tcp23 [8/256/0/180] [ 7287.998651] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7288.016992] LNet: Added LNI 192.168.202.144@tcp24 [8/256/0/180] [ 7290.352927] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7290.361408] LNet: Added LNI 192.168.202.144@tcp25 [8/256/0/180] [ 7292.163456] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7292.182597] LNet: Added LNI 192.168.202.144@tcp26 [8/256/0/180] [ 7294.532878] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7294.548955] LNet: Added LNI 192.168.202.144@tcp27 [8/256/0/180] [ 7296.433173] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7296.445191] LNet: Added LNI 192.168.202.144@tcp28 [8/256/0/180] [ 7298.935970] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7298.955575] LNet: Added LNI 192.168.202.144@tcp29 [8/256/0/180] [ 7301.031890] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7301.041571] LNet: Added LNI 192.168.202.144@tcp30 [8/256/0/180] [ 7303.586053] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7303.598166] LNet: Added LNI 192.168.202.144@tcp31 [8/256/0/180] [ 7305.778196] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7305.786871] LNet: Added LNI 192.168.202.144@tcp32 [8/256/0/180] [ 7308.054838] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7308.072765] LNet: Added LNI 192.168.202.144@tcp33 [8/256/0/180] [ 7310.610957] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7310.619673] LNet: Added LNI 192.168.202.144@tcp34 [8/256/0/180] [ 7312.756329] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7312.767667] LNet: Added LNI 192.168.202.144@tcp35 [8/256/0/180] [ 7315.374152] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7315.393158] LNet: Added LNI 192.168.202.144@tcp36 [8/256/0/180] [ 7317.707415] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7317.723470] LNet: Added LNI 192.168.202.144@tcp37 [8/256/0/180] [ 7320.536169] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7320.553822] LNet: Added LNI 192.168.202.144@tcp38 [8/256/0/180] [ 7322.877275] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7322.895046] LNet: Added LNI 192.168.202.144@tcp39 [8/256/0/180] [ 7325.769861] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7325.778818] LNet: Added LNI 192.168.202.144@tcp40 [8/256/0/180] [ 7327.900700] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7327.909462] LNet: Added LNI 192.168.202.144@tcp41 [8/256/0/180] [ 7330.529778] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7330.545503] LNet: Added LNI 192.168.202.144@tcp42 [8/256/0/180] [ 7333.020424] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7333.035694] LNet: Added LNI 192.168.202.144@tcp43 [8/256/0/180] [ 7336.163014] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7336.171839] LNet: Added LNI 192.168.202.144@tcp44 [8/256/0/180] [ 7338.527254] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7338.545694] LNet: Added LNI 192.168.202.144@tcp45 [8/256/0/180] [ 7341.505307] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7341.513881] LNet: Added LNI 192.168.202.144@tcp46 [8/256/0/180] [ 7344.744051] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7344.763751] LNet: Added LNI 192.168.202.144@tcp47 [8/256/0/180] [ 7347.535418] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7347.555305] LNet: Added LNI 192.168.202.144@tcp48 [8/256/0/180] [ 7350.999523] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7351.008791] LNet: Added LNI 192.168.202.144@tcp49 [8/256/0/180] [ 7353.552216] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7353.570722] LNet: Added LNI 192.168.202.144@tcp50 [8/256/0/180] [ 7363.557390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7363.559146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7363.565552] Lustre: Skipped 2 previous similar messages [ 7363.572079] Lustre: Skipped 2 previous similar messages [ 7368.245567] Lustre: server umount lustre-MDT0000 complete [ 7370.030852] LustreError: 443386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775891405 with bad export cookie 1000390358022095066 [ 7370.037891] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7370.038196] LustreError: 443386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7370.177552] Lustre: server umount lustre-MDT0001 complete [ 7379.109296] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 7382.397341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.176219] Lustre: 442667:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775891409/real 1775891409] req@0000000031c80fbc x1862156957078016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775891425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7390.196095] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7392.224109] Lustre: 442668:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775891411/real 1775891411] req@00000000e46a2a7a x1862156957078144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775891427 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7394.272177] Lustre: 442669:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775891414/real 1775891414] req@00000000762bc5ee x1862156957078272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775891430 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7394.336419] LustreError: 442665:0:(client.c:1393:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000041ccd60 x1862156957079168/t0(0) o250->MGC192.168.202.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7394.485140] LustreError: 445131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7394.491485] LustreError: 445131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7394.501681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7395.553670] LustreError: 445132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7396.148281] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7398.562368] LustreError: 445131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7399.616359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7399.785462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7400.928239] Lustre: 442667:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775891420/real 1775891420] req@000000001a21b5ac x1862156957078656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775891436 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7400.937939] Lustre: 442667:0:(client.c:2482:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7401.385312] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7403.400467] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7405.040770] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7405.169186] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7406.048225] Lustre: 442668:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775891425/real 1775891425] req@00000000ea10e097 x1862156957078784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775891441 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7415.269543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7415.271272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7415.273132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7415.276783] LustreError: Skipped 1 previous similar message [ 7415.284656] Lustre: Skipped 1 previous similar message [ 7420.391125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7420.395316] Lustre: Skipped 2 previous similar messages [ 7420.503799] Lustre: server umount lustre-OST0000 complete [ 7425.507931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7425.508434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7425.518054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7425.528979] Lustre: Skipped 1 previous similar message [ 7428.462866] Lustre: server umount lustre-MDT0000 complete [ 7430.316931] LustreError: 467343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775891466 with bad export cookie 1000390358022097145 [ 7430.323226] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7430.324466] LustreError: 467343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7430.473069] Lustre: server umount lustre-MDT0001 complete [ 7435.051432] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 7436.383838] Key type lgssc unregistered [ 7436.534180] LNet: 471073:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7436.536684] LNetError: 471073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7436.538534] LNet: Removed LNI 192.168.202.144@tcp [ 7436.539731] LNet: Removed LNI 192.168.202.144@tcp10 [ 7436.540805] LNet: Removed LNI 192.168.202.144@tcp11 [ 7436.541825] LNet: Removed LNI 192.168.202.144@tcp12 [ 7436.542830] LNet: Removed LNI 192.168.202.144@tcp13 [ 7436.543898] LNet: Removed LNI 192.168.202.144@tcp14 [ 7436.544902] LNet: Removed LNI 192.168.202.144@tcp15 [ 7436.545935] LNet: Removed LNI 192.168.202.144@tcp16 [ 7436.546961] LNet: Removed LNI 192.168.202.144@tcp17 [ 7436.547929] LNet: Removed LNI 192.168.202.144@tcp18 [ 7436.548942] LNet: Removed LNI 192.168.202.144@tcp19 [ 7436.549959] LNet: Removed LNI 192.168.202.144@tcp20 [ 7436.550977] LNet: Removed LNI 192.168.202.144@tcp21 [ 7436.551930] LNet: Removed LNI 192.168.202.144@tcp22 [ 7436.552939] LNet: Removed LNI 192.168.202.144@tcp23 [ 7436.554018] LNet: Removed LNI 192.168.202.144@tcp24 [ 7436.554950] LNet: Removed LNI 192.168.202.144@tcp25 [ 7436.555938] LNet: Removed LNI 192.168.202.144@tcp26 [ 7436.556934] LNet: Removed LNI 192.168.202.144@tcp27 [ 7436.557937] LNet: Removed LNI 192.168.202.144@tcp28 [ 7436.558984] LNet: Removed LNI 192.168.202.144@tcp29 [ 7436.559991] LNet: Removed LNI 192.168.202.144@tcp30 [ 7436.560959] LNet: Removed LNI 192.168.202.144@tcp31 [ 7436.562017] LNet: Removed LNI 192.168.202.144@tcp32 [ 7436.562982] LNet: Removed LNI 192.168.202.144@tcp33 [ 7436.564054] LNet: Removed LNI 192.168.202.144@tcp34 [ 7436.564995] LNet: Removed LNI 192.168.202.144@tcp35 [ 7436.565946] LNet: Removed LNI 192.168.202.144@tcp36 [ 7436.566981] LNet: Removed LNI 192.168.202.144@tcp37 [ 7436.567940] LNet: Removed LNI 192.168.202.144@tcp38 [ 7436.568930] LNet: Removed LNI 192.168.202.144@tcp39 [ 7436.569966] LNet: Removed LNI 192.168.202.144@tcp40 [ 7436.570957] LNet: Removed LNI 192.168.202.144@tcp41 [ 7436.571972] LNet: Removed LNI 192.168.202.144@tcp42 [ 7436.572984] LNet: Removed LNI 192.168.202.144@tcp43 [ 7436.573944] LNet: Removed LNI 192.168.202.144@tcp44 [ 7436.574951] LNet: Removed LNI 192.168.202.144@tcp45 [ 7436.575943] LNet: Removed LNI 192.168.202.144@tcp46 [ 7436.576936] LNet: Removed LNI 192.168.202.144@tcp47 [ 7436.577947] LNet: Removed LNI 192.168.202.144@tcp48 [ 7436.578951] LNet: Removed LNI 192.168.202.144@tcp49 [ 7436.585360] LNet: Removed LNI 192.168.202.144@tcp50 [ 7436.811539] Key type .llcrypt unregistered [ 7436.812494] Key type ._llcrypt unregistered [ 7445.505261] Key type ._llcrypt registered [ 7445.507291] Key type .llcrypt registered [ 7445.559045] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 7445.856662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7445.862583] alg: No test for adler32 (adler32-zlib) [ 7446.710218] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 7446.801823] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 7448.384122] Key type lgssc registered [ 7448.823648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7452.496983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7452.502525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7453.666664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7455.368307] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7459.030268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.230317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7461.027572] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7463.178904] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7464.914183] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7467.914936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7468.019031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7470.393916] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7472.409554] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7478.258416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7487.384631] loop0: detected capacity change from 0 to 204800000 [ 7487.476770] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7487.480249] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7487.484866] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7490.363071] loop0: detected capacity change from 0 to 204800000 [ 7490.454335] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7490.458348] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7490.466588] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7492.953308] loop0: detected capacity change from 0 to 204800000 [ 7495.563936] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7495.645520] Lustre: fs15246-OST0000: new disk, initializing [ 7495.649504] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7495.681634] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7498.238935] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7503.125376] loop1: detected capacity change from 0 to 204800000 [ 7505.504418] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.567685] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7505.609749] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7505.659582] Lustre: fs15246-MDT0000: new disk, initializing [ 7505.750739] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7505.764656] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7505.833076] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7506.849251] Lustre: 476678:0:(client.c:1640:after_reply()) @@@ resending request on EINPROGRESS req@00000000222e0ee4 x1862157186664320/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1775891558 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7507.639901] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7515.939738] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7515.945896] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7515.962500] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7517.166761] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7517.174330] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7522.285156] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7523.378251] Lustre: server umount fs15246-MDT0000 complete [ 7527.325765] Lustre: server umount fs15246-OST0000 complete [ 7532.521107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7532.522580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7532.526073] Lustre: Skipped 1 previous similar message [ 7532.532260] Lustre: Skipped 1 previous similar message [ 7537.639805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7537.643797] Lustre: Skipped 1 previous similar message [ 7537.774706] Lustre: server umount lustre-OST0000 complete [ 7541.217546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7541.222444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7542.755068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7542.758899] Lustre: Skipped 2 previous similar messages [ 7545.471543] Lustre: server umount lustre-MDT0000 complete [ 7547.289917] LustreError: 472275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775891583 with bad export cookie 7810520534340457968 [ 7547.295174] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7547.296649] LustreError: 472275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7547.461548] Lustre: server umount lustre-MDT0001 complete [ 7551.627397] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 7553.186607] Key type lgssc unregistered [ 7553.356669] LNet: 479737:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7553.361494] LNetError: 479737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7553.375702] LNet: Removed LNI 192.168.202.144@tcp [ 7553.840908] Key type .llcrypt unregistered [ 7553.843287] Key type ._llcrypt unregistered [ 7563.218906] Key type ._llcrypt registered [ 7563.220578] Key type .llcrypt registered [ 7563.274260] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 7563.651164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7563.660749] alg: No test for adler32 (adler32-zlib) [ 7564.530385] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 7564.630094] LNet: Added LNI 192.168.202.144@tcp [8/256/0/180] [ 7566.232174] Key type lgssc registered [ 7566.837848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7584.666725] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 7589.469134] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 7606.538772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7606.551699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7607.678209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7607.696497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7607.758516] Lustre: lustre-MDT0000: new disk, initializing [ 7607.825221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7607.837327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7609.705968] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7615.482283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.559294] Lustre: 484126:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7615.609097] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7615.612859] Lustre: Skipped 1 previous similar message [ 7615.666087] Lustre: lustre-MDT0001: new disk, initializing [ 7615.729606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7615.755609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7615.763788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7617.563953] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7621.476898] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7623.266464] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7626.579309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7626.736303] Lustre: lustre-OST0000: new disk, initializing [ 7626.739465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7626.783095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7629.333581] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_default_debug -1 all [ 7633.316676] Lustre: DEBUG MARKER: oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7634.437182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7634.441353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7634.477091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7635.037304] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7635.157320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7636.925271] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7637.043822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7639.525056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7639.527751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7639.529851] LustreError: Skipped 1 previous similar message [ 7639.538695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7639.544253] Lustre: Skipped 1 previous similar message [ 7644.076054] Lustre: server umount lustre-OST0000 complete [ 7646.689505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7646.694429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7646.702169] Lustre: Skipped 1 previous similar message [ 7646.705577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7649.762093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7649.763148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7651.602950] Lustre: server umount lustre-MDT0000 complete [ 7652.987432] LustreError: 484903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775891688 with bad export cookie 3762817216521072157 [ 7652.991667] LustreError: 484903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7652.993689] LustreError: MGC192.168.202.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7653.139495] Lustre: server umount lustre-MDT0001 complete [ 7662.843502] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing set_hostid [ 7667.053058] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing load_modules_local [ 7680.103422] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7451 sec ========== 03:15:15 (1775891715) [ 7680.881917] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:15:15 (1775891715) === [ 7682.494407] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:15:17 (1775891717) === [ 7697.405324] Lustre: DEBUG MARKER: oleg244-server.virtnet: executing unload_modules_local [ 7698.281868] Key type lgssc unregistered [ 7698.397333] LNet: 492302:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7698.400992] LNetError: 492302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7698.410261] LNet: Removed LNI 192.168.202.144@tcp [ 7698.668083] Key type .llcrypt unregistered [ 7698.669048] Key type ._llcrypt unregistered