[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 607455960 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2889096K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002250] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006812] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007047] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009160] LSM: Security Framework initializing [ 0.010086] Yama: becoming mindful. [ 0.011045] SELinux: Initializing. [ 0.012124] *** VALIDATE selinux *** [ 0.022178] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028498] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030099] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031149] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032222] *** VALIDATE tmpfs *** [ 0.034231] *** VALIDATE proc *** [ 0.035367] *** VALIDATE cgroup *** [ 0.036013] *** VALIDATE cgroup2 *** [ 0.037317] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038182] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039016] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041019] Spectre V2 : User space: Vulnerable [ 0.042029] Speculative Store Bypass: Vulnerable [ 0.045285] debug: unmapping init [mem 0xffffffff95259000-0xffffffff95260fff] [ 0.048219] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049950] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051040] ... version: 2 [ 0.052017] ... bit width: 48 [ 0.053016] ... generic registers: 4 [ 0.054015] ... value mask: 0000ffffffffffff [ 0.055025] ... max period: 00007fffffffffff [ 0.056017] ... fixed-purpose events: 3 [ 0.057011] ... event mask: 000000070000000f [ 0.058423] rcu: Hierarchical SRCU implementation. [ 0.060863] smp: Bringing up secondary CPUs ... [ 0.061705] x86: Booting SMP configuration: [ 0.062034] .... node #0, CPUs: #1 #2 #3 [ 0.068313] smp: Brought up 1 node, 4 CPUs [ 0.070021] smpboot: Max logical packages: 1 [ 0.071019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144031] node 0 deferred pages initialised in 70ms [ 0.147241] devtmpfs: initialized [ 0.148357] x86/mm: Memory block size: 128MB [ 0.152232] gcov: version magic: 0x41383552 [ 0.155318] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159141] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162461] pinctrl core: initialized pinctrl subsystem [ 0.164227] [ 0.165008] ************************************************************* [ 0.168021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171019] ** ** [ 0.173021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176014] ** ** [ 0.178011] ** This means that this kernel is built to expose internal ** [ 0.181015] ** IOMMU data structures, which may compromise security on ** [ 0.183013] ** your system. ** [ 0.186014] ** ** [ 0.188018] ** If you see this message and you are not debugging the ** [ 0.190015] ** kernel, report this immediately to your vendor! ** [ 0.193026] ** ** [ 0.195013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198014] ************************************************************* [ 0.200850] NET: Registered protocol family 16 [ 0.203548] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.206067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.208094] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.213014] cpuidle: using governor menu [ 0.215082] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.216653] PCI: Using configuration type 1 for base access [ 0.217154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224098] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231111] cryptd: max_cpu_qlen set to 1000 [ 0.234313] ACPI: Added _OSI(Module Device) [ 0.235008] ACPI: Added _OSI(Processor Device) [ 0.236011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.237018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242121] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248347] ACPI: Interpreter enabled [ 0.250148] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251020] ACPI: Using IOAPIC for interrupt routing [ 0.253370] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257488] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.268879] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281465] acpiphp: Slot [2] registered [ 0.283180] acpiphp: Slot [5] registered [ 0.285167] acpiphp: Slot [6] registered [ 0.287208] acpiphp: Slot [7] registered [ 0.288215] acpiphp: Slot [8] registered [ 0.290109] acpiphp: Slot [9] registered [ 0.291256] acpiphp: Slot [10] registered [ 0.293186] acpiphp: Slot [3] registered [ 0.295210] acpiphp: Slot [4] registered [ 0.297149] acpiphp: Slot [11] registered [ 0.298172] acpiphp: Slot [12] registered [ 0.300127] acpiphp: Slot [13] registered [ 0.301202] acpiphp: Slot [14] registered [ 0.303211] acpiphp: Slot [15] registered [ 0.305171] acpiphp: Slot [16] registered [ 0.307144] acpiphp: Slot [17] registered [ 0.308308] acpiphp: Slot [18] registered [ 0.310153] acpiphp: Slot [19] registered [ 0.312142] acpiphp: Slot [20] registered [ 0.313143] acpiphp: Slot [21] registered [ 0.315112] acpiphp: Slot [22] registered [ 0.317238] acpiphp: Slot [23] registered [ 0.318194] acpiphp: Slot [24] registered [ 0.320111] acpiphp: Slot [25] registered [ 0.322178] acpiphp: Slot [26] registered [ 0.324145] acpiphp: Slot [27] registered [ 0.325178] acpiphp: Slot [28] registered [ 0.327180] acpiphp: Slot [29] registered [ 0.329180] acpiphp: Slot [30] registered [ 0.330099] acpiphp: Slot [31] registered [ 0.333122] PCI host bridge to bus 0000:00 [ 0.334025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.336029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.339034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.341027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347216] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.351000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.354250] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.364817] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.371112] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.377037] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381580] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383889] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.386050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389900] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.414882] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.420120] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.429025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.435028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.474035] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.500290] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.508026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.519028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.544020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.555761] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.569032] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.580035] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.609036] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.626417] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.633028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.637024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.652014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.661121] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.666041] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.673027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.691037] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.703184] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.720034] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.730029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.752026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.782366] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.785488] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.790488] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.793397] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.796218] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.802290] iommu: Default domain type: Passthrough [ 0.804519] SCSI subsystem initialized [ 0.807224] ACPI: bus type USB registered [ 0.808152] usbcore: registered new interface driver usbfs [ 0.809188] usbcore: registered new interface driver hub [ 0.811090] usbcore: registered new device driver usb [ 0.812198] pps_core: LinuxPPS API ver. 1 registered [ 0.814022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.816085] PTP clock support registered [ 0.818151] EDAC MC: Ver: 3.0.0 [ 0.820220] PCI: Using ACPI for IRQ routing [ 0.822456] NetLabel: Initializing [ 0.824018] NetLabel: domain hash size = 128 [ 0.825017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.827124] NetLabel: unlabeled traffic allowed by default [ 0.830186] vgaarb: loaded [ 0.832372] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.834015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.839402] clocksource: Switched to clocksource kvm-clock [ 0.958473] VFS: Disk quotas dquot_6.6.0 [ 0.960344] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.963119] *** VALIDATE ramfs *** [ 0.964429] *** VALIDATE hugetlbfs *** [ 0.966183] pnp: PnP ACPI init [ 0.968735] pnp: PnP ACPI: found 6 devices [ 0.992301] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.997124] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.999902] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.002949] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.005609] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.009559] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.013401] NET: Registered protocol family 2 [ 1.016904] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.022644] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.026278] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.031771] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.036087] TCP: Hash tables configured (established 65536 bind 65536) [ 1.039270] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.042439] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.045464] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.049201] NET: Registered protocol family 1 [ 1.052343] RPC: Registered named UNIX socket transport module. [ 1.055092] RPC: Registered udp transport module. [ 1.056897] RPC: Registered tcp transport module. [ 1.059441] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.062631] NET: Registered protocol family 44 [ 1.064427] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.067510] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.070558] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.074803] PCI: CLS 0 bytes, default 64 [ 1.077524] Unpacking initramfs... [ 2.522369] debug: unmapping init [mem 0xffff95d9fcc54000-0xffff95d9fffbffff] [ 2.526119] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.527908] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.530155] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.011728] Initialise system trusted keyrings [ 3.013378] Key type blacklist registered [ 3.015375] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.023972] zbud: loaded [ 3.027984] *** VALIDATE nfs *** [ 3.029459] *** VALIDATE nfs4 *** [ 3.031265] pstore: using deflate compression [ 3.034842] Platform Keyring initialized [ 3.135604] NET: Registered protocol family 38 [ 3.136862] Key type asymmetric registered [ 3.138077] Asymmetric key parser 'x509' registered [ 3.139953] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.142225] io scheduler mq-deadline registered [ 3.143808] io scheduler kyber registered [ 3.145636] io scheduler bfq registered [ 3.147310] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.149947] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.152881] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.155048] ACPI: Power Button [PWRF] [ 3.158778] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.164407] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.175776] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.183875] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.198337] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.224879] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.253215] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.258680] Non-volatile memory driver v1.3 [ 3.259992] Linux agpgart interface v0.103 [ 3.288835] virtio_blk virtio1: [vda] 145920 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.291601] vda: detected capacity change from 0 to 74711040 [ 3.305261] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.308181] vdb: detected capacity change from 0 to 1073741824 [ 3.322787] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.325669] vdc: detected capacity change from 0 to 2621440000 [ 3.339100] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.341916] vdd: detected capacity change from 0 to 2621440000 [ 3.356759] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.359560] vde: detected capacity change from 0 to 4294967296 [ 3.381277] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.384219] vdf: detected capacity change from 0 to 4294967296 [ 3.390464] libphy: Fixed MDIO Bus: probed [ 3.397272] usbcore: registered new interface driver usbserial_generic [ 3.400084] usbserial: USB Serial support registered for generic [ 3.402485] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.406474] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.407928] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.410622] mousedev: PS/2 mouse device common for all mice [ 3.413623] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.416751] rtc_cmos 00:05: RTC can wake from S4 [ 3.420113] rtc_cmos 00:05: registered as rtc0 [ 3.422207] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.422275] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.425140] intel_pstate: CPU model not supported [ 3.430189] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.434385] hid: raw HID events driver (C) Jiri Kosina [ 3.437084] usbcore: registered new interface driver usbhid [ 3.439580] usbhid: USB HID core driver [ 3.441458] drop_monitor: Initializing network drop monitor service [ 3.443564] Initializing XFRM netlink socket [ 3.445085] NET: Registered protocol family 10 [ 3.447276] Segment Routing with IPv6 [ 3.448390] NET: Registered protocol family 17 [ 3.450180] mpls_gso: MPLS GSO support [ 3.456851] RAS: Correctable Errors collector initialized. [ 3.459074] AVX version of gcm_enc/dec engaged. [ 3.460448] AES CTR mode by8 optimization enabled [ 3.541066] sched_clock: Marking stable (3541032328, 0)->(4496760888, -955728560) [ 3.545202] registered taskstats version 1 [ 3.547673] Loading compiled-in X.509 certificates [ 3.549537] zswap: loaded using pool lzo/zbud [ 3.572925] Key type big_key registered [ 3.585655] Key type encrypted registered [ 3.587444] ima: No TPM chip found, activating TPM-bypass! [ 3.589304] ima: Allocated hash algorithm: sha1 [ 3.590766] ima: No architecture policies found [ 3.592531] evm: Initialising EVM extended attributes: [ 3.594287] evm: security.selinux [ 3.595556] evm: security.ima [ 3.596702] evm: security.capability [ 3.598102] evm: HMAC attrs: 0x1 [ 3.600462] rtc_cmos 00:05: setting system clock to 2026-08-19 05:50:32 UTC (1787118632) [ 3.605910] debug: unmapping init [mem 0xffffffff96203000-0xffffffff963fffff] [ 3.609183] debug: unmapping init [mem 0xffffffff94f82000-0xffffffff95258fff] [ 3.618068] Write protecting the kernel read-only data: 28672k [ 3.620229] debug: unmapping init [mem 0xffffffff93603000-0xffffffff937fffff] [ 3.622233] debug: unmapping init [mem 0xffffffff93f14000-0xffffffff93ffffff] [ 3.657693] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.665161] systemd[1]: Detected virtualization kvm. [ 3.666795] systemd[1]: Detected architecture x86-64. [ 3.668387] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.695228] systemd[1]: No hostname configured. [ 3.696763] systemd[1]: Set hostname to . [ 3.698712] random: systemd: uninitialized urandom read (16 bytes read) [ 3.700813] systemd[1]: Initializing machine ID from random generator. [ 3.735417] random: ln: uninitialized urandom read (6 bytes read) [ 3.837179] random: systemd: uninitialized urandom read (16 bytes read) [ 3.840147] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.845880] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.851748] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.546461] device-mapper: uevent: version 1.0.3 [ 4.548703] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [[ 5.396439] random: fast init done  OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.454764] virtio_net virtio0 ens2: renamed from eth0 [ 5.590311] scsi host0: ata_piix [ 5.624477] scsi host1: ata_piix [ 5.627893] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.630675] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.066519] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.103882] random: crng init done [ 10.105533] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.794874] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.108793] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.393949] SELinux: Disabled at runtime. [ 12.456458] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.464278] systemd[1]: Detected virtualization kvm. [ 12.466156] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.799756] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.805384] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.828153] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.836314] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.840669] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.850316] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.866320] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ 14.176898] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 15.930090] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.865403] hrtimer: interrupt took 5489426 ns [ 17.084119] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.741450] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.122445] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.238113] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 22.687525] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 23.565193] NFS: Registering the id_resolver key type [ 23.568111] Key type id_resolver registered [ 23.570654] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started OpenSSH server daemon. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg251-server login: [ 99.300053] libcfs: loading out-of-tree module taints kernel. [ 99.390656] Key type ._llcrypt registered [ 99.395830] Key type .llcrypt registered [ 99.653873] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 124.717928] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 126.948956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 126.980365] alg: No test for adler32 (adler32-zlib) [ 128.382881] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 128.906465] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 130.631229] Key type lgssc registered [ 133.242262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 154.951375] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 202.545496] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 218.148529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 218.207201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.631716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 219.688370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 219.766851] Lustre: lustre-MDT0000: new disk, initializing [ 219.836407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.856447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 225.235866] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.547224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 239.679037] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 239.706381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 239.709628] Lustre: Skipped 1 previous similar message [ 239.787203] Lustre: lustre-MDT0001: new disk, initializing [ 239.846257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 239.882594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 239.895250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 244.154629] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.039865] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 260.177378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.598243] Lustre: lustre-OST0000: new disk, initializing [ 260.605740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 260.613453] Lustre: 8456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.696792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 266.313461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 266.321660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 266.386843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 268.771719] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 283.255814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 283.454463] Lustre: lustre-OST0001: new disk, initializing [ 283.456369] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 283.464719] Lustre: 9529:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 283.524324] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 289.517647] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.408792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 291.416535] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 291.464788] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 301.037361] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 308.572803] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 314.713069] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing check_logdir /tmp/testlogs/ [ 320.318907] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing yml_node [ 324.480690] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 326.858850] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 329.006660] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 330.655375] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 01:55:57 EDT 2026 [ 349.637504] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 351.608224] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 368.095776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 368.102965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.119527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 368.124059] Lustre: Skipped 3 previous similar messages [ 368.144900] Lustre: Skipped 3 previous similar messages [ 373.222705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 374.114048] Lustre: server umount lustre-MDT0000 complete [ 378.005061] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119006 with bad export cookie 2713276284155131947 [ 378.006626] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 378.021377] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 378.336470] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.339832] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 378.354326] Lustre: Skipped 1 previous similar message [ 378.381117] Lustre: Skipped 4 previous similar messages [ 383.456547] LustreError: 6527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.457737] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 383.472606] LustreError: 6527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 383.497856] Lustre: Skipped 1 previous similar message [ 384.547146] Lustre: server umount lustre-MDT0001 complete [ 389.044740] Lustre: server umount lustre-OST0000 complete [ 392.965985] Lustre: server umount lustre-OST0001 complete [ 399.344527] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 408.381623] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 441.079346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.305227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 441.334473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 441.392385] Lustre: lustre-MDT0000: new disk, initializing [ 441.458186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.473615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 446.306591] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 456.596772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.743649] Lustre: 15084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 456.805589] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 456.809139] Lustre: Skipped 1 previous similar message [ 456.869972] Lustre: lustre-MDT0001: new disk, initializing [ 456.948127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 456.974017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 456.978708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 460.792434] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 469.949264] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 476.552978] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 484.165352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 484.599911] Lustre: lustre-OST0000: new disk, initializing [ 484.604379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 484.609749] Lustre: 17054:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 484.763686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 486.274605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 486.282883] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 486.361796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 490.746489] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 501.525780] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 505.739968] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 505.950735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 510.319610] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 510.528623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 517.111965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 517.112975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.131141] LustreError: Skipped 1 previous similar message [ 517.132538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 517.142015] Lustre: Skipped 1 previous similar message [ 517.166229] Lustre: Skipped 1 previous similar message [ 518.763792] Lustre: server umount lustre-OST0000 complete [ 523.744905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 523.750741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.333883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 527.356470] Lustre: Skipped 2 previous similar messages [ 528.801882] Lustre: server umount lustre-MDT0000 complete [ 532.449661] LustreError: 15097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.466337] LustreError: 15097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 533.140087] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119162 with bad export cookie 2713276284155133487 [ 533.146233] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.164835] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 533.506843] Lustre: server umount lustre-MDT0001 complete [ 537.891731] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:59:24 (1787119164) [ 540.411861] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 542.629757] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:59:28 (1787119168) [ 550.496834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.957261] LustreError: 19463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.047531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.065420] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 556.526072] LustreError: 19464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.565894] LustreError: 19464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 560.612416] LustreError: 19463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.261962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.978569] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 577.752722] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 583.750375] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 590.455407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 598.806118] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 608.043980] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 620.512525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 620.524295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.531069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 620.536159] Lustre: Skipped 2 previous similar messages [ 625.773870] Lustre: server umount lustre-OST0000 complete [ 630.752740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.760916] Lustre: Skipped 1 previous similar message [ 636.504865] Lustre: server umount lustre-MDT0000 complete [ 640.333553] LustreError: 20197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119269 with bad export cookie 2713276284155134369 [ 640.335859] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.344334] LustreError: 20197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 640.694571] Lustre: server umount lustre-MDT0001 complete [ 653.315779] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 656.571393] Key type lgssc unregistered [ 656.854873] LNet: 23012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.860182] LNetError: 23012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 657.899954] LNet: Removed LNI 192.168.202.151@tcp [ 659.181300] Key type .llcrypt unregistered [ 659.183081] Key type ._llcrypt unregistered [ 685.089857] Key type ._llcrypt registered [ 685.091608] Key type .llcrypt registered [ 685.199225] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 686.411795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 686.574441] alg: No test for adler32 (adler32-zlib) [ 687.667715] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 687.871517] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 689.543232] Key type lgssc registered [ 690.930901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 699.836106] loop: module loaded [ 699.922392] loop0: detected capacity change from 0 to 204800000 [ 700.194819] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.218677] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.484620] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 705.723665] loop0: detected capacity change from 0 to 204800000 [ 706.003160] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 706.020076] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 706.029267] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 712.836879] loop0: detected capacity change from 0 to 204800000 [ 718.852698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 718.875770] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 720.137603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 720.191328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 720.326317] Lustre: lustre-MDT0000: new disk, initializing [ 720.586176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.622630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 726.749245] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 737.201332] loop1: detected capacity change from 0 to 204800000 [ 744.223669] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 744.496831] Lustre: lustre-OST07c6: new disk, initializing [ 744.500612] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 744.504265] Lustre: Skipped 1 previous similar message [ 744.510658] Lustre: 26211:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 744.615352] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 750.754076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 750.762106] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 750.804856] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 751.079587] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 754.930469] Lustre: 25199:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 754.939042] Lustre: 25199:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 754.952538] Lustre: 25199:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 754.967661] Lustre: 25199:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 754.979927] Lustre: 25199:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 754.995630] Lustre: 25199:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 771.108284] Lustre: 25199:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 771.139455] Lustre: 25199:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 771.148290] Lustre: 25199:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 771.160092] Lustre: 25199:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 771.168475] Lustre: 25199:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 771.174129] Lustre: 25199:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 776.520822] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 776.528650] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 776.534438] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 776.540965] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.547292] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 776.555959] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.571187] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 776.583646] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.592930] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 776.597639] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.602379] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 776.606686] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.875864] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 779.885955] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 779.891461] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 779.899979] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.905227] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 779.910801] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.914778] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 779.919623] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.923757] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 779.929704] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.933492] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 779.936597] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.278879] Lustre: 26219:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 785.294483] Lustre: 26219:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 785.302583] Lustre: 26219:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 785.313294] Lustre: 26219:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.319711] Lustre: 26219:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.324600] Lustre: 26219:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.328879] Lustre: 26219:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 785.334475] Lustre: 26219:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.344143] Lustre: 26219:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 785.352840] Lustre: 26219:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.359752] Lustre: 26219:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.365321] Lustre: 26219:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 796.159938] Lustre: 25199:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 796.169575] Lustre: 25199:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 796.175748] Lustre: 25199:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 796.184429] Lustre: 25199:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 796.198438] Lustre: 25199:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 796.202902] Lustre: 25199:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 796.226431] Lustre: 25199:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 796.242193] Lustre: 25199:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 796.253604] Lustre: 25199:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 796.267467] Lustre: 25199:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 796.276449] Lustre: 25199:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 796.280515] Lustre: 25199:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 843.235475] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 843.244152] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 843.261860] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 848.366830] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 848.566149] Lustre: server umount lustre-OST07c6 complete [ 857.417535] Lustre: server umount lustre-MDT0000 complete [ 874.910435] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 877.538791] Key type lgssc unregistered [ 877.862768] LNet: 28768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.871796] LNetError: 28768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.906351] LNet: Removed LNI 192.168.202.151@tcp [ 878.761289] Key type .llcrypt unregistered [ 878.764891] Key type ._llcrypt unregistered [ 897.941918] Key type ._llcrypt registered [ 897.944949] Key type .llcrypt registered [ 898.133545] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 899.720799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 899.900198] alg: No test for adler32 (adler32-zlib) [ 901.762757] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 901.997661] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 903.672604] Key type lgssc registered [ 904.586414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 913.109882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 913.126583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.433501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.269258] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 924.644039] LustreError: 30042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 924.672504] LustreError: 30042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 926.162302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.572092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 930.275744] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 936.333872] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 940.889730] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 947.462506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 947.833864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.178895] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 959.158827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 961.100076] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 977.890154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 977.897432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.910258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 979.940173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 979.955472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.964179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 979.985513] Lustre: Skipped 1 previous similar message [ 982.480134] Lustre: server umount lustre-OST0000 complete [ 988.128948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 988.142141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.164544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 990.184260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 990.186412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 992.096262] Lustre: server umount lustre-MDT0000 complete [ 995.297655] LustreError: 32126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.313826] LustreError: 32126:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 996.118864] LustreError: 30021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119625 with bad export cookie 5489639101434167801 [ 996.132491] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 996.607967] Lustre: server umount lustre-MDT0001 complete [ 1006.651970] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1010.260139] Key type lgssc unregistered [ 1010.547339] LNet: 33784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.558772] LNetError: 33784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.597303] LNet: Removed LNI 192.168.202.151@tcp [ 1011.751939] Key type .llcrypt unregistered [ 1011.755063] Key type ._llcrypt unregistered [ 1032.421871] Key type ._llcrypt registered [ 1032.430651] Key type .llcrypt registered [ 1032.592665] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1033.682772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1033.697407] alg: No test for adler32 (adler32-zlib) [ 1034.750753] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 1035.005666] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1036.695182] Key type lgssc registered [ 1037.731532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1045.327301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1045.346571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1046.903269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.837215] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1056.238201] LustreError: 35057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1056.260134] LustreError: 35057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1060.758484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.291248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1066.697313] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1075.063156] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1079.997032] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1086.661674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1086.971287] LustreError: 36877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1087.125198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1091.275370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1094.314602] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1102.111598] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1112.035530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1112.040714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.054658] LustreError: Skipped 1 previous similar message [ 1112.086789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.159638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.177669] Lustre: Skipped 2 previous similar messages [ 1118.479218] Lustre: server umount lustre-OST0000 complete [ 1122.272573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.277703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1122.294905] Lustre: Skipped 2 previous similar messages [ 1122.308642] Lustre: Skipped 1 previous similar message [ 1127.392223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1127.399746] Lustre: Skipped 1 previous similar message [ 1128.506520] Lustre: server umount lustre-MDT0000 complete [ 1132.048432] LustreError: 35036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119760 with bad export cookie 4153626490832698466 [ 1132.068708] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1132.395511] Lustre: server umount lustre-MDT0001 complete [ 1143.398300] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1146.197547] Key type lgssc unregistered [ 1146.448466] LNet: 38591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1146.450944] LNetError: 38591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1147.496362] LNet: Removed LNI 192.168.202.151@tcp [ 1148.373775] Key type .llcrypt unregistered [ 1148.376786] Key type ._llcrypt unregistered [ 1173.661919] Key type ._llcrypt registered [ 1173.666158] Key type .llcrypt registered [ 1173.795396] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1174.975868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1175.064171] alg: No test for adler32 (adler32-zlib) [ 1176.331974] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 1176.609848] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1178.279152] Key type lgssc registered [ 1179.545648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1187.418665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1187.432259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.981373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1194.139671] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1198.570793] LustreError: 40009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.602788] LustreError: 40009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1202.947188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.370307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1208.661279] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1217.129977] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1222.944169] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1229.846362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1230.384270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1233.467849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1238.133559] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1246.472183] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1253.870194] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.877269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1253.891257] Lustre: Skipped 2 previous similar messages [ 1253.918796] Lustre: Skipped 2 previous similar messages [ 1254.640773] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 1255.407486] Lustre: server umount lustre-MDT0000 complete [ 1258.980335] LustreError: 40005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1259.004283] LustreError: 40005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1259.302819] LustreError: 39990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119888 with bad export cookie 9035784392949234122 [ 1259.327152] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1259.751768] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [ 1260.063350] LustreError: 40739:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000002dd4ad6 x1873930936676736/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1264.096211] LustreError: 40005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.097522] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.127171] LustreError: 40005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1264.155427] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1265.784754] Lustre: server umount lustre-MDT0001 complete [ 1295.327163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1295.504191] Lustre: server umount lustre-OST0000 complete [ 1312.841416] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1315.844962] Key type lgssc unregistered [ 1316.162739] LNet: 43877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.179987] LNetError: 43877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1316.204642] LNet: Removed LNI 192.168.202.151@tcp [ 1317.207788] Key type .llcrypt unregistered [ 1317.209986] Key type ._llcrypt unregistered [ 1341.409575] Key type ._llcrypt registered [ 1341.411860] Key type .llcrypt registered [ 1341.511741] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1343.130367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1343.236399] alg: No test for adler32 (adler32-zlib) [ 1344.392503] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 1344.641902] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1346.295347] Key type lgssc registered [ 1347.498614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1355.374520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1355.413984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1356.909333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.726497] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1366.500046] LustreError: 45294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1366.514698] LustreError: 45294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1369.677779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1369.935689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1376.475056] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1386.506951] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1391.952224] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1399.960557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1400.228444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1401.266207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1406.336754] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1413.779956] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1419.746322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.767345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1419.770402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1420.521957] Lustre: lustre-OST0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 1420.529606] Lustre: Skipped 1 previous similar message [ 1424.864353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1425.628186] LustreError: 47115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.897730] Lustre: server umount lustre-OST0000 complete [ 1447.021541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.048173] Lustre: Skipped 1 previous similar message [ 1447.057957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1447.075367] Lustre: Skipped 2 previous similar messages [ 1453.387217] Lustre: server umount lustre-MDT0000 complete [ 1454.566787] LustreError: 45294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.596994] LustreError: 45294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1457.359609] LustreError: 45276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120086 with bad export cookie 17378769399988399454 [ 1457.378547] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1457.391166] LustreError: 45276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1464.187537] Lustre: server umount lustre-MDT0001 complete [ 1475.229740] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1479.208428] Key type lgssc unregistered [ 1479.469147] LNet: 49006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.475535] LNetError: 49006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.498235] LNet: Removed LNI 192.168.202.151@tcp [ 1480.529107] Key type .llcrypt unregistered [ 1480.531765] Key type ._llcrypt unregistered [ 1505.607587] Key type ._llcrypt registered [ 1505.611115] Key type .llcrypt registered [ 1505.777800] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1507.614497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1507.732918] alg: No test for adler32 (adler32-zlib) [ 1508.777972] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 1509.090307] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1510.823245] Key type lgssc registered [ 1512.146413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1521.506040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1521.537783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1523.019933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.597459] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1532.385323] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.418761] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1537.512248] LustreError: 50424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1537.664297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1537.936357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1542.553914] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1551.021628] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1559.523039] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1566.820541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1567.255688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1568.316193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1573.421131] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1580.078805] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1588.441909] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1591.562240] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1604.008107] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-19 2h17m09s [ 1605.914354] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1608.161585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.162570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1608.168278] Lustre: Skipped 1 previous similar message [ 1608.175872] Lustre: Skipped 2 previous similar messages [ 1609.188172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.201294] Lustre: Skipped 2 previous similar messages [ 1612.524529] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [ 1614.472501] Lustre: server umount lustre-MDT0000 complete [ 1615.336385] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1615.371818] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1618.486989] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1619.659571] LustreError: 52913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1624.634888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.765803] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1625.075322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1625.143050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1626.128057] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1629.822194] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1632.557714] Lustre: DEBUG MARKER: Wait for df (36727) ... [ 1634.067534] Lustre: DEBUG MARKER: done [ 1638.882451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1638.888772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1638.903339] Lustre: Skipped 1 previous similar message [ 1638.907673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1638.914134] Lustre: Skipped 1 previous similar message [ 1640.416346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1640.423074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.542040] Lustre: server umount lustre-OST0000 complete [ 1651.680557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.681694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1651.708293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1651.716489] Lustre: Skipped 4 previous similar messages [ 1653.354929] Lustre: server umount lustre-MDT0000 complete [ 1657.415394] LustreError: 50405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120286 with bad export cookie 18043595156794744000 [ 1657.424565] LustreError: 50405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1657.426438] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1657.808283] Lustre: server umount lustre-MDT0001 complete [ 1667.290570] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1669.849487] Key type lgssc unregistered [ 1670.122927] LNet: 55351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.134650] LNetError: 55351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1670.146750] LNet: Removed LNI 192.168.202.151@tcp [ 1671.058573] Key type .llcrypt unregistered [ 1671.064124] Key type ._llcrypt unregistered [ 1712.848607] Key type ._llcrypt registered [ 1712.851057] Key type .llcrypt registered [ 1712.990730] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1714.546788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1714.592065] alg: No test for adler32 (adler32-zlib) [ 1715.662864] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 1715.945245] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1717.639437] Key type lgssc registered [ 1718.684675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1725.671901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1725.687045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1726.913092] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1726.941484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1727.123691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.945700] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1738.757170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.807256] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1738.829957] Lustre: 57571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1739.010799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1742.919931] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1749.640928] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1754.743501] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1760.403560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1760.662863] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1760.847619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1766.668223] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1769.092485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1772.747332] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1779.398244] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-19 2h20m05s [ 1780.813449] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1786.485894] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1788.418980] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1788.464779] LustreError: 59724:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1808.612561] Lustre: lustre-MDT0000: Client efd54715-f00a-4303-a5eb-f64b6f3dbbb7 (at 192.168.202.51@tcp) reconnecting [ 1828.551301] LustreError: 59724:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1830.432942] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1832.670525] Lustre: DEBUG MARKER: done [ 1839.013698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.015053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1839.022979] Lustre: Skipped 1 previous similar message [ 1839.035376] Lustre: Skipped 1 previous similar message [ 1840.933137] Lustre: server umount lustre-OST0000 complete [ 1846.752766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1846.763980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.798510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1847.841073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1847.846893] Lustre: Skipped 2 previous similar messages [ 1850.957446] Lustre: server umount lustre-MDT0000 complete [ 1854.800574] LustreError: 57563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120483 with bad export cookie 11728626072585270445 [ 1854.829760] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1855.260618] Lustre: server umount lustre-MDT0001 complete [ 1865.228358] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1867.996685] Key type lgssc unregistered [ 1868.229987] LNet: 61901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1868.245874] LNetError: 61901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.285936] LNet: Removed LNI 192.168.202.151@tcp [ 1870.075447] Key type .llcrypt unregistered [ 1870.077614] Key type ._llcrypt unregistered [ 1911.208101] Key type ._llcrypt registered [ 1911.210921] Key type .llcrypt registered [ 1911.356703] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1912.798922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1913.015941] alg: No test for adler32 (adler32-zlib) [ 1914.313354] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 1914.671567] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1916.367485] Key type lgssc registered [ 1917.665724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1920.556171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1920.586365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.804973] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1921.846691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1921.997993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.011662] Lustre: Failing over lustre-MDT0000 [ 1923.294025] Lustre: server umount lustre-MDT0000 complete [ 1930.184862] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:22:36 (1787120556) [ 1937.309411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.834167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.829206] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1951.740957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.844348] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1951.875915] Lustre: 64700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1952.257726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1956.855508] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1964.141949] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1970.093427] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1976.173941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1976.444129] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1976.607718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.932835] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1984.158900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1988.179478] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1992.662404] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1998.322280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1998.329197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.584514] Lustre: server umount lustre-MDT0000 complete [ 1999.846872] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1999.859634] Lustre: Skipped 1 previous similar message [ 2002.282766] LustreError: 64693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120631 with bad export cookie 10617039494327421304 [ 2002.296518] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2002.312150] LustreError: 64693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2002.691907] Lustre: server umount lustre-MDT0001 complete [ 2006.022921] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2010.543503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2018.628960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.255574] Lustre: 63624:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120633/real 1787120633] req@000000001f8fa8a7 x1873931710533888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787120649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2020.279607] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.327511] Lustre: 63626:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120636/real 1787120636] req@0000000060eaa6fd x1873931710534016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787120652 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2025.439480] Lustre: 63626:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120638/real 1787120638] req@0000000015d00a5e x1873931710534144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787120654 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2027.833337] LustreError: 66550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.861031] LustreError: 66550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2027.956387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.034640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2028.959093] Lustre: 63624:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120641/real 1787120641] req@00000000f97e6d4f x1873931710534400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787120657 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2031.031580] LustreError: 68468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2031.056434] LustreError: 68468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2032.683584] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2035.040791] Lustre: 63623:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120647/real 1787120647] req@0000000066996bc3 x1873931710534528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787120663 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2036.193978] LustreError: 68466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.207536] LustreError: 68466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2039.285359] LustreError: 68467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.307682] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2039.319451] LustreError: 68467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2041.831075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.239659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2047.510191] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2047.743123] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2056.101858] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2062.014802] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2067.939262] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.941305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2067.958577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2070.361598] Lustre: server umount lustre-MDT0000 complete [ 2073.056837] LustreError: 68467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.086244] LustreError: 68467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2074.056925] LustreError: 64692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120702 with bad export cookie 10617039494327423579 [ 2074.058971] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2074.069478] LustreError: 64692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2074.466986] Lustre: server umount lustre-MDT0001 complete [ 2080.741848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2090.139971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2094.559340] Lustre: 63625:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120707/real 1787120707] req@000000001f8fa8a7 x1873931710561024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787120723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2094.588178] Lustre: 63625:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2094.596919] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.607320] Lustre: Skipped 2 previous similar messages [ 2098.720734] LustreError: 63622:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001d6a8772 x1873931710561920/t0(0) o250->MGC192.168.202.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2099.083479] LustreError: 69202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.190979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2099.279851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2104.124773] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2112.110158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.897504] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2117.620176] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2123.815087] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2124.163685] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2129.764376] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2138.084739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2138.101085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.114522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2138.117126] Lustre: Skipped 2 previous similar messages [ 2139.116121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.125433] Lustre: Skipped 2 previous similar messages [ 2144.224185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.347911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.350727] Lustre: Skipped 4 previous similar messages [ 2150.367123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2150.532690] Lustre: server umount lustre-MDT0000 complete [ 2154.183126] LustreError: 64693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120783 with bad export cookie 10617039494327424958 [ 2154.185136] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2154.192016] LustreError: 64693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2154.467406] LustreError: 71148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.469581] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.488475] LustreError: 71148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2154.522889] Lustre: Skipped 2 previous similar messages [ 2154.650810] Lustre: server umount lustre-MDT0001 complete [ 2160.534460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2163.967909] Lustre: DEBUG MARKER: files compared the same [ 2165.764861] Lustre: server umount lustre-OST0000 complete [ 2180.426723] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2182.745891] Key type lgssc unregistered [ 2182.909557] LNet: 74426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.912871] LNetError: 74426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.923263] LNet: Removed LNI 192.168.202.151@tcp [ 2183.455073] Key type .llcrypt unregistered [ 2183.456895] Key type ._llcrypt unregistered [ 2207.477871] Key type ._llcrypt registered [ 2207.479438] Key type .llcrypt registered [ 2207.685378] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2209.533409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2209.551061] alg: No test for adler32 (adler32-zlib) [ 2210.665486] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 2210.947098] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2212.718414] Key type lgssc registered [ 2213.967528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2221.838707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2221.857099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.369952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.645463] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug +malloc all [ 2233.833262] LustreError: 75845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.849577] LustreError: 75845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2235.969261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.296200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2241.172602] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug +malloc all [ 2248.167166] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2253.206455] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2259.752437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.041781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.037175] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug +malloc all [ 2268.277191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2273.673967] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2282.978396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2282.980857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2283.008465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2284.004192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2284.005017] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.005274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2287.754399] Lustre: server umount lustre-OST0000 complete [ 2292.713077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2292.721645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.737308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.744288] Lustre: Skipped 1 previous similar message [ 2297.716979] Lustre: server umount lustre-MDT0000 complete [ 2299.364387] LustreError: 75844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.370736] LustreError: 75844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2302.304767] LustreError: 75823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120931 with bad export cookie 4927961570421188610 [ 2302.308311] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2302.310369] LustreError: 75823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2302.524475] Lustre: server umount lustre-MDT0001 complete [ 2315.822225] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2319.408952] Key type lgssc unregistered [ 2319.802268] LNet: 79378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2319.824237] LNetError: 79378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.870622] LNet: Removed LNI 192.168.202.151@tcp [ 2322.021195] Key type .llcrypt unregistered [ 2322.026980] Key type ._llcrypt unregistered [ 2349.585973] Key type ._llcrypt registered [ 2349.587368] Key type .llcrypt registered [ 2349.707203] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2350.826385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2350.912784] alg: No test for adler32 (adler32-zlib) [ 2351.969674] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 2352.218358] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2353.895346] Key type lgssc registered [ 2355.354629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2363.389382] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2363.422358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2379.039483] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2384.159740] LustreError: 80771:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.151@tcp: failed processing log, type 4: rc = -110 [ 2410.847522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2418.318343] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2429.068140] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2438.107300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.249072] Lustre: *** cfs_fail_loc=706, val=0*** [ 2438.636251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2438.710936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2442.947073] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2443.744327] LustreError: 80797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.770632] LustreError: 80797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2445.799663] LustreError: 80796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.815311] LustreError: 80796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2450.916685] LustreError: 80797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.929757] LustreError: 80797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2452.021738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.490030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2457.391652] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2465.208280] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2470.289463] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2472.930328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2472.932885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2472.937987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2472.938870] Lustre: Skipped 1 previous similar message [ 2472.951437] Lustre: Skipped 1 previous similar message [ 2478.061173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2478.070993] Lustre: Skipped 1 previous similar message [ 2478.580245] Lustre: server umount lustre-OST0000 complete [ 2488.289767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.291398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2488.311764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2488.330429] Lustre: Skipped 1 previous similar message [ 2490.511510] Lustre: server umount lustre-MDT0000 complete [ 2493.408855] LustreError: 81863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.421490] LustreError: 81863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2494.919942] LustreError: 80779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121123 with bad export cookie 13967802201882521274 [ 2494.920882] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.927250] LustreError: 80779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2495.383823] Lustre: server umount lustre-MDT0001 complete [ 2507.041261] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2510.455929] Key type lgssc unregistered [ 2510.778141] LNet: 84392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.784817] LNetError: 84392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2510.809960] LNet: Removed LNI 192.168.202.151@tcp [ 2511.799805] Key type .llcrypt unregistered [ 2511.803211] Key type ._llcrypt unregistered [ 2538.763356] Key type ._llcrypt registered [ 2538.764787] Key type .llcrypt registered [ 2538.949786] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2540.215344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2540.588633] alg: No test for adler32 (adler32-zlib) [ 2541.714422] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 2542.103323] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2543.831462] Key type lgssc registered [ 2545.770386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2557.007149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2557.032365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.350799] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2574.271739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.263601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2580.452027] LustreError: 86497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.475533] LustreError: 86497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2580.748341] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2584.548496] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.577421] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2589.669403] LustreError: 86500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.711805] LustreError: 86500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2593.071123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2593.804955] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.846534] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2593.861474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.881478] LustreError: 87243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.902577] LustreError: 87243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2599.963634] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2609.413463] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2609.424032] Lustre: MGS: Client 0db192fa-0ad7-4ba1-a50b-4526ce6e87b6 (at 0@lo) reconnecting [ 2609.445000] LustreError: 86511:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2609.805540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.930273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2615.283706] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2631.648535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2631.659270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.670380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2635.235512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2635.242074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2635.264154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2636.883643] Lustre: server umount lustre-OST0000 complete [ 2640.864525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2640.872483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2640.876911] LustreError: 85167:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000c7b7a02 x1873932367984000/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2640.882807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2640.882817] Lustre: Skipped 1 previous similar message [ 2640.886145] Lustre: Skipped 1 previous similar message [ 2645.988081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2645.990766] Lustre: Skipped 1 previous similar message [ 2647.253580] Lustre: server umount lustre-MDT0000 complete [ 2651.105899] LustreError: 87243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.127697] LustreError: 87243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2652.301642] Lustre: server umount lustre-MDT0001 complete [ 2656.381218] Lustre: server umount MGS complete [ 2670.157304] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:34:56 (1787121296) [ 2702.441836] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 2713.800755] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2751.738096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2756.658912] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2766.994859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.346350] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2767.429800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2767.538659] Lustre: lustre-MDT0001: new disk, initializing [ 2767.636108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2767.709588] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2767.728688] Lustre: 93691:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2767.741802] LustreError: 93691:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2772.233846] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2783.917452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2784.252322] Lustre: lustre-OST0000: new disk, initializing [ 2784.259358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2784.266989] Lustre: 94457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2784.348815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.154756] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2792.498419] Lustre: 94831:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000015d00a5e x1873932367996288/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787121437 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2800.679700] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2808.897136] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2808.911936] Lustre: MGS: Client 2b8bc821-55de-4ce1-8dfc-f00bbb395f00 (at 0@lo) reconnecting [ 2808.985824] Lustre: 92959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2809.007256] LustreError: 93677:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2809.058828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2809.152909] Lustre: lustre-MDT0000: new disk, initializing [ 2809.346415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2809.385891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2809.413065] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2810.468469] Lustre: 95525:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000aecc207f x1873932368003328/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787121455 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2815.686427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2815.709788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2815.770457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2816.630633] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2819.628681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2819.636914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2845.670688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2845.682313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2845.699678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2850.234846] Lustre: server umount lustre-OST0000 complete [ 2856.928879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2856.932461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.953675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2856.962778] Lustre: Skipped 2 previous similar messages [ 2856.975106] Lustre: Skipped 3 previous similar messages [ 2860.769956] Lustre: server umount lustre-MDT0000 complete [ 2862.054130] LustreError: 95952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.070060] LustreError: 95952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2871.463491] Lustre: server umount lustre-MDT0001 complete [ 2876.124681] Lustre: server umount MGS complete [ 2890.874634] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:38:36 (1787121516) [ 2915.999798] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 2919.493781] Key type lgssc unregistered [ 2919.826330] LNet: 98027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2919.835554] LNetError: 98027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2919.857677] LNet: Removed LNI 192.168.202.151@tcp [ 2920.972509] Key type .llcrypt unregistered [ 2920.974353] Key type ._llcrypt unregistered [ 2942.913278] Key type ._llcrypt registered [ 2942.916258] Key type .llcrypt registered [ 2943.030686] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2944.532861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2944.703576] alg: No test for adler32 (adler32-zlib) [ 2946.042861] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 2946.418165] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 2948.215358] Key type lgssc registered [ 2949.469333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2959.254969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2959.280717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2960.945517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2966.186625] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2971.113060] LustreError: 99252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.135866] LustreError: 99252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2975.981541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.360062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2981.679572] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2990.536586] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2996.900544] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3004.265981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3004.596138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3006.652329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3011.252530] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3019.886946] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3032.544820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3032.551838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.563329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3032.567564] Lustre: Skipped 1 previous similar message [ 3037.353391] Lustre: server umount lustre-OST0000 complete [ 3042.804284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.813277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3042.819280] Lustre: Skipped 1 previous similar message [ 3042.830382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3042.840275] LustreError: Skipped 1 previous similar message [ 3042.847680] Lustre: Skipped 1 previous similar message [ 3047.741974] Lustre: server umount lustre-MDT0000 complete [ 3047.909128] LustreError: 99252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3047.930888] LustreError: 99252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3052.311560] LustreError: 99233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121681 with bad export cookie 3163905057849940463 [ 3052.316562] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3052.323580] LustreError: 99233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3052.698916] Lustre: server umount lustre-MDT0001 complete [ 3065.078454] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3068.569242] Key type lgssc unregistered [ 3069.089509] LNet: 102794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3069.107192] LNetError: 102794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3070.184959] LNet: Removed LNI 192.168.202.151@tcp [ 3071.498434] Key type .llcrypt unregistered [ 3071.500792] Key type ._llcrypt unregistered [ 3090.303368] Key type ._llcrypt registered [ 3090.305233] Key type .llcrypt registered [ 3090.478429] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3092.028777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3092.144559] alg: No test for adler32 (adler32-zlib) [ 3093.460440] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 3093.749506] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3095.471896] Key type lgssc registered [ 3096.782861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3104.003781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3104.017181] LustreError: 103870:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3104.098248] LustreError: 103880:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3104.106498] LustreError: 103870:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4920 [ 3104.120820] LustreError: 103870:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3104.133033] LustreError: 103870:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3104.135657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.891303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3114.036455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.328354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3119.307398] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3126.301353] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing lsmod [ 3129.932926] LustreError: 105078:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3129.983558] LustreError: 105085:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3129.990911] LustreError: 105078:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4951 [ 3130.001685] LustreError: 105078:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3130.004166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3130.007381] LustreError: 105078:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3130.491464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3131.514601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3134.957799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3134.971505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.992702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3136.999060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3137.004417] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.019182] Lustre: Skipped 1 previous similar message [ 3139.109298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.127753] Lustre: Skipped 4 previous similar messages [ 3140.690755] Lustre: server umount lustre-MDT0000 complete [ 3144.163828] LustreError: 103917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3144.186778] LustreError: 103917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3145.655313] LustreError: 103896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121774 with bad export cookie 10182939245384802063 [ 3145.665260] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3145.667427] LustreError: 103896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3147.359332] LustreError: 104348:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000dbc2af48 x1873932946790528/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3147.891385] Lustre: server umount lustre-MDT0001 complete [ 3163.618684] Lustre: server umount lustre-OST0000 complete [ 3173.974679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.582780] LustreError: 106150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3174.655368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.002723] LustreError: 106151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3180.480791] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3185.129633] LustreError: 106150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.250187] LustreError: 106151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.474530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.048566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3196.675689] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3206.374581] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3212.914992] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3219.697617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3220.184091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3221.257497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:97) [ 3226.678637] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3235.072765] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3246.048514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3246.051826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.072592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3246.085288] Lustre: Skipped 1 previous similar message [ 3246.108472] Lustre: Skipped 1 previous similar message [ 3251.172424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3251.186221] Lustre: Skipped 1 previous similar message [ 3256.288314] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3256.517790] Lustre: server umount lustre-OST0000 complete [ 3261.413276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3261.422871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3261.431021] Lustre: Skipped 1 previous similar message [ 3261.442253] Lustre: Skipped 3 previous similar messages [ 3266.964407] Lustre: server umount lustre-MDT0000 complete [ 3271.359756] LustreError: 106132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121900 with bad export cookie 10182939245384802980 [ 3271.368815] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3271.386273] LustreError: 106132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3271.965762] Lustre: server umount lustre-MDT0001 complete [ 3287.313968] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3290.758823] Key type lgssc unregistered [ 3291.180575] LNet: 109693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3291.183848] LNetError: 109693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3291.202860] LNet: Removed LNI 192.168.202.151@tcp [ 3292.167669] Key type .llcrypt unregistered [ 3292.170575] Key type ._llcrypt unregistered [ 3322.461330] Key type ._llcrypt registered [ 3322.462868] Key type .llcrypt registered [ 3322.565345] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3323.650097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3323.770038] alg: No test for adler32 (adler32-zlib) [ 3324.939762] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 3325.285353] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3326.944873] Key type lgssc registered [ 3328.425795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3338.218920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3338.238214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3339.812241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.447896] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3349.985477] LustreError: 111112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.998347] LustreError: 111112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3355.105940] LustreError: 111113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.762377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3356.289616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3361.484673] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3369.955904] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3376.320086] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3384.182423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3384.316651] LustreError: 112935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3384.745525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3389.954756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3391.585032] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3400.102980] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3406.464459] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3409.737850] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3415.520519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3415.525679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.540920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3416.555412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3416.557114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3416.557918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.600602] Lustre: Skipped 1 previous similar message [ 3421.670759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3421.688191] Lustre: Skipped 1 previous similar message [ 3426.798736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3426.810417] Lustre: Skipped 1 previous similar message [ 3429.343198] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3429.484546] Lustre: server umount lustre-OST0000 complete [ 3437.024453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3437.038873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3437.047480] Lustre: Skipped 1 previous similar message [ 3437.062881] Lustre: Skipped 1 previous similar message [ 3440.139501] Lustre: server umount lustre-MDT0000 complete [ 3442.144474] LustreError: 111108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3442.155477] LustreError: 111108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3444.031447] LustreError: 111094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122072 with bad export cookie 8505969910665039194 [ 3444.041592] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3444.041742] LustreError: 111094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3444.620335] Lustre: server umount lustre-MDT0001 complete [ 3457.351757] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3460.892366] Key type lgssc unregistered [ 3461.326671] LNet: 114755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3461.333262] LNetError: 114755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3461.360294] LNet: Removed LNI 192.168.202.151@tcp [ 3462.344759] Key type .llcrypt unregistered [ 3462.346686] Key type ._llcrypt unregistered [ 3482.721699] Key type ._llcrypt registered [ 3482.724233] Key type .llcrypt registered [ 3482.825278] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3484.476377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3484.531768] alg: No test for adler32 (adler32-zlib) [ 3485.751416] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 3486.090808] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3487.887218] Key type lgssc registered [ 3489.628375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3497.978555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3498.032212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.683715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3505.201737] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3509.220583] LustreError: 115975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.258045] LustreError: 115975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3514.341700] LustreError: 115976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3514.387435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.742480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3519.637709] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3528.629781] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3534.533829] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3542.100902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3542.508064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3545.604794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3549.061414] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3557.580184] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3570.166872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3570.170337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3570.204957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3571.168845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3571.170032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3571.199316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3575.034876] Lustre: server umount lustre-OST0000 complete [ 3581.409751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3581.428583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3581.440163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3581.446780] Lustre: Skipped 1 previous similar message [ 3581.453357] Lustre: Skipped 1 previous similar message [ 3586.529134] LustreError: 117466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3586.545316] LustreError: 117466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3586.818654] Lustre: server umount lustre-MDT0000 complete [ 3590.925960] LustreError: 115956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122219 with bad export cookie 99978797104591464 [ 3590.939360] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3590.942807] LustreError: 115956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3591.634577] Lustre: server umount lustre-MDT0001 complete [ 3604.632309] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3607.575932] Key type lgssc unregistered [ 3607.832306] LNet: 119513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3607.839275] LNetError: 119513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3607.856864] LNet: Removed LNI 192.168.202.151@tcp [ 3609.043749] Key type .llcrypt unregistered [ 3609.045752] Key type ._llcrypt unregistered [ 3635.837722] Key type ._llcrypt registered [ 3635.840717] Key type .llcrypt registered [ 3635.959576] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3637.370681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3637.382861] alg: No test for adler32 (adler32-zlib) [ 3638.463838] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 3638.693766] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3640.479239] Key type lgssc registered [ 3641.657931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3650.103486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3650.135569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3651.713269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3657.134414] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3661.281746] LustreError: 120941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.289849] LustreError: 120941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3666.400632] LustreError: 120942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3666.702925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3667.106138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3671.889573] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3680.018336] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3686.079580] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3692.488837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3692.919086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3697.025887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 3699.092720] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3705.031788] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3715.533308] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3719.440879] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3719.452576] Lustre: Skipped 2 previous similar messages [ 3723.394900] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3723.437925] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3724.219079] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3724.249306] Lustre: Skipped 2 previous similar messages [ 3729.029641] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3729.042539] Lustre: Skipped 1 previous similar message [ 3735.182079] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.51@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3735.190993] Lustre: Skipped 2 previous similar messages [ 3738.768205] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.51@tcp 192.168.0.[2,10]@tcp [ 3743.315442] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.51/24@tcp in log lustre-MDT0000 [ 3743.326560] Lustre: Skipped 2 previous similar messages [ 3745.756323] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.51/24@tcp [ 3745.778535] Lustre: Skipped 1 previous similar message [ 3751.927854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3751.938205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.947228] LustreError: Skipped 1 previous similar message [ 3751.992291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3752.427623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3752.436796] LustreError: Skipped 4 previous similar messages [ 3752.440813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.046941] Lustre: server umount lustre-OST0000 complete [ 3759.072328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3759.086959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3759.099233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3759.101551] Lustre: Skipped 2 previous similar messages [ 3762.657143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3762.657529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3765.354879] Lustre: server umount lustre-MDT0000 complete [ 3767.788470] LustreError: 124340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3767.813966] LustreError: 124340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3769.894229] LustreError: 120922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122398 with bad export cookie 11663648739700521593 [ 3769.895987] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3769.908574] LustreError: 120922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3770.301938] Lustre: server umount lustre-MDT0001 complete [ 3781.420862] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3784.660748] Key type lgssc unregistered [ 3784.901417] LNet: 125743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3784.908520] LNetError: 125743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3784.926204] LNet: Removed LNI 192.168.202.151@tcp [ 3785.704349] Key type .llcrypt unregistered [ 3785.706317] Key type ._llcrypt unregistered [ 3813.923370] Key type ._llcrypt registered [ 3813.929613] Key type .llcrypt registered [ 3814.123297] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3815.884532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3815.909317] alg: No test for adler32 (adler32-zlib) [ 3817.031155] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 3817.343915] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3819.064096] Key type lgssc registered [ 3820.229561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3827.794348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3827.812883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.353629] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3829.374548] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.51/24@tcp [ 3829.393665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3833.705613] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3839.983592] LustreError: 127317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3840.013317] LustreError: 127317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3841.530592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3841.876443] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3841.907422] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.51/24@tcp [ 3841.979907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3845.814629] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3852.964411] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3860.297664] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3866.744519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3866.929458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3870.971350] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3875.151839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:201 to 0x240000400:225) [ 3876.936324] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3896.804264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3896.814181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3896.827673] Lustre: Skipped 1 previous similar message [ 3896.854223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3896.854925] Lustre: Skipped 1 previous similar message [ 3901.929371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3901.952543] Lustre: Skipped 1 previous similar message [ 3902.299935] Lustre: server umount lustre-OST0000 complete [ 3907.052145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.064570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3907.071032] Lustre: Skipped 1 previous similar message [ 3907.078123] Lustre: Skipped 1 previous similar message [ 3912.164631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3912.173267] Lustre: Skipped 1 previous similar message [ 3913.009803] Lustre: server umount lustre-MDT0000 complete [ 3916.774299] LustreError: 127299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122545 with bad export cookie 11584255766805842571 [ 3916.781820] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3917.151682] Lustre: server umount lustre-MDT0001 complete [ 3926.867465] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3930.486905] Key type lgssc unregistered [ 3930.866893] LNet: 130927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3930.873815] LNetError: 130927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3930.899956] LNet: Removed LNI 192.168.202.151@tcp [ 3931.708169] Key type .llcrypt unregistered [ 3931.714375] Key type ._llcrypt unregistered [ 3958.591164] Key type ._llcrypt registered [ 3958.593121] Key type .llcrypt registered [ 3958.666607] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 3978.786404] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3981.167408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3981.208503] alg: No test for adler32 (adler32-zlib) [ 3982.478878] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 3982.890975] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3984.583154] Key type lgssc registered [ 3985.769609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4019.740664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4019.766111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4020.960337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4020.986578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4021.041183] Lustre: lustre-MDT0000: new disk, initializing [ 4021.090931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4021.103284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4025.720971] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4037.272441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4037.473167] Lustre: 134328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4037.586565] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4037.595333] Lustre: Skipped 1 previous similar message [ 4037.659141] Lustre: lustre-MDT0001: new disk, initializing [ 4037.767406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4037.825361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4037.835908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4042.657139] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4053.176656] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4058.338422] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4064.343252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4064.643630] Lustre: lustre-OST0000: new disk, initializing [ 4064.650779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4064.655954] Lustre: 136296:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4064.781737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4071.832798] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4075.108180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4075.120818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4075.259165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4082.746498] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4088.791967] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4088.954412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4095.028723] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4095.210240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4101.091530] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4101.094687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4101.097802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4101.103752] LustreError: Skipped 1 previous similar message [ 4101.115630] Lustre: Skipped 1 previous similar message [ 4103.642516] Lustre: server umount lustre-OST0000 complete [ 4109.281062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4109.285995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.300060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.314307] Lustre: Skipped 1 previous similar message [ 4111.332974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4111.337178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4111.351332] Lustre: Skipped 1 previous similar message [ 4116.710365] Lustre: server umount lustre-MDT0000 complete [ 4121.287860] LustreError: 134319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122750 with bad export cookie 8654114763880829667 [ 4121.295905] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4121.307087] LustreError: 134319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4121.920402] Lustre: server umount lustre-MDT0001 complete [ 4131.157045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4131.687078] LustreError: 138516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4131.709588] LustreError: 138516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4131.777127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4136.729808] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4136.942828] LustreError: 138517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.054837] LustreError: 138516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4145.864592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4146.335525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4151.187463] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4159.718928] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4165.794872] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4174.231609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4174.664510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.779757] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4191.800928] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4195.864292] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4196.028649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4200.198986] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4200.458262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4209.589503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4209.846152] Lustre: lustre-OST0001: new disk, initializing [ 4209.854837] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4209.865308] Lustre: 141889:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4210.007385] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4218.562098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4218.577159] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4218.771369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4219.338105] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4228.543247] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4228.928710] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4236.816411] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4237.232656] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4244.975232] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a4f118dd800.ost_server_uuid 50 [ 4246.688315] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a4f118dd800.ost_server_uuid in IDLE state after 0 sec [ 4257.129889] Lustre: server umount lustre-OST0001 complete [ 4259.817396] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.837225] LustreError: 140339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.851897] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4263.905493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4263.914618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.917480] LustreError: Skipped 1 previous similar message [ 4263.924482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4263.940792] Lustre: Skipped 2 previous similar messages [ 4264.934277] LustreError: 141008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.960294] LustreError: 141008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4267.956442] Lustre: server umount lustre-OST0000 complete [ 4274.146601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4274.153279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.162612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4274.170691] Lustre: Skipped 3 previous similar messages [ 4278.582466] Lustre: server umount lustre-MDT0000 complete [ 4280.291586] LustreError: 139244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.330085] LustreError: 139244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4282.503695] LustreError: 139251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122911 with bad export cookie 8654114763880830549 [ 4282.513586] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4282.521925] LustreError: 139251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4282.953058] Lustre: server umount lustre-MDT0001 complete [ 4304.543783] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 4309.103860] Key type lgssc unregistered [ 4309.459588] LNet: 144908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4309.464730] LNetError: 144908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4309.478689] LNet: Removed LNI 192.168.202.151@tcp [ 4310.433441] Key type .llcrypt unregistered [ 4310.435180] Key type ._llcrypt unregistered [ 4357.402338] Key type ._llcrypt registered [ 4357.404666] Key type .llcrypt registered [ 4357.534503] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 4374.677954] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4376.254972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4376.280316] alg: No test for adler32 (adler32-zlib) [ 4377.199674] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 4377.360597] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 4378.959767] Key type lgssc registered [ 4380.117885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4422.143483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4422.166041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4423.348584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4423.382490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4423.459958] Lustre: lustre-MDT0000: new disk, initializing [ 4423.529146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.559816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4428.615730] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4440.250304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.352386] Lustre: 149068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4440.411923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4440.415121] Lustre: Skipped 1 previous similar message [ 4440.511644] Lustre: lustre-MDT0001: new disk, initializing [ 4440.656706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4440.703245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4440.713321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4445.594135] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4455.459571] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4462.366854] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4471.090447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4471.530483] Lustre: lustre-OST0000: new disk, initializing [ 4471.548563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4471.561490] Lustre: 151037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4471.717257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.244807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4481.267413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4481.462974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4481.864113] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4492.374501] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4496.371386] Lustre: Failing over lustre-OST0000 [ 4496.612658] Lustre: server umount lustre-OST0000 complete [ 4512.226378] Lustre: 147289:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787123125/real 1787123125] req@00000000c0e8e649 x1873934293174656/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787123141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4512.241033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.246185] Lustre: 147289:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4512.269495] Lustre: Skipped 1 previous similar message [ 4514.314065] Lustre: 147292:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787123127/real 1787123127] req@0000000010612c3a x1873934293174912/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787123143 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4514.351158] Lustre: 147292:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4514.730917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4515.060033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4515.079842] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4515.587256] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4518.117247] Lustre: 147291:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787123130/real 1787123130] req@0000000090c35906 x1873934293175808/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787123146 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4518.143358] Lustre: 147291:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4520.847380] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4522.210503] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4522.212338] Lustre: lustre-OST0000: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [ 4523.296536] Lustre: 147289:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787123136/real 1787123136] req@00000000f68cc91e x1873934293177088/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787123152 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4524.367407] Lustre: Failing over lustre-MDT0000 [ 4524.605261] Lustre: server umount lustre-MDT0000 complete [ 4527.597196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4527.599617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.617033] Lustre: Skipped 2 previous similar messages [ 4530.899726] LustreError: 149074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4530.919521] LustreError: 149074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4532.704880] LustreError: 149078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4532.739550] LustreError: 149078:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4536.070958] LustreError: 149073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.385056] Lustre: 149072:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787123153/real 1787123153] req@00000000a024d91a x1873934293186432/t0(0) o101->MGC192.168.202.151@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1787123169 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4540.454183] Lustre: 149072:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4540.471417] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4540.501806] Lustre: 149072:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4540.503469] LustreError: 149079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.542433] LustreError: 149079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4545.508601] LustreError: 149078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.543733] LustreError: 149078:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4546.055053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4550.933638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4551.012370] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4551.886588] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4556.289260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4556.297430] Lustre: Skipped 1 previous similar message [ 4556.325979] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4556.384036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4556.718708] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4566.513298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4566.516413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4566.539556] Lustre: Skipped 1 previous similar message [ 4571.627970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4571.640849] Lustre: Skipped 2 previous similar messages [ 4572.409320] Lustre: server umount lustre-OST0000 complete [ 4576.738206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4576.782232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4576.785054] Lustre: Skipped 1 previous similar message [ 4581.860601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4581.878966] Lustre: Skipped 1 previous similar message [ 4582.567529] Lustre: server umount lustre-MDT0000 complete [ 4586.980029] LustreError: 149073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4587.004048] LustreError: 149073:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4587.109818] LustreError: 149862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123216 with bad export cookie 10949970415972173640 [ 4587.112961] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4587.117934] LustreError: 149862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4587.545613] Lustre: server umount lustre-MDT0001 complete [ 4600.962365] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 4604.381621] Key type lgssc unregistered [ 4604.811737] LNet: 154698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4604.823688] LNetError: 154698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4604.861092] LNet: Removed LNI 192.168.202.151@tcp [ 4605.900964] Key type .llcrypt unregistered [ 4605.907845] Key type ._llcrypt unregistered [ 4646.305202] Key type ._llcrypt registered [ 4646.307581] Key type .llcrypt registered [ 4646.394331] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4647.800823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4647.832303] alg: No test for adler32 (adler32-zlib) [ 4648.900923] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 4649.119325] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 4650.775164] Key type lgssc registered [ 4652.220886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4660.735925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4660.758025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4662.424329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4666.630748] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4672.994641] LustreError: 156218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4673.016325] LustreError: 156218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4676.492191] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4678.116662] LustreError: 156219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4680.993475] LustreError: 156247:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4681.002122] LustreError: 156247:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4681.013913] LustreError: 156247:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4681.886497] Lustre: server umount lustre-MDT0000 complete [ 4699.571511] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 4712.038604] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4751.393311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.743977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4751.772169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4751.909725] Lustre: lustre-MDT0000: new disk, initializing [ 4752.003644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.022581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4757.503424] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4769.176315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4769.308667] Lustre: 159550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4769.363969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4769.372855] Lustre: Skipped 1 previous similar message [ 4769.454319] Lustre: lustre-MDT0001: new disk, initializing [ 4769.598388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4769.674200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4769.694958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4775.378410] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4785.656805] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4792.171424] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4799.851088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4800.198153] Lustre: lustre-OST0000: new disk, initializing [ 4800.205497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4800.222048] Lustre: 161522:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4800.376561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4805.721649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4805.742697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4805.919874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4808.225569] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4817.991569] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4822.900236] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4824.833323] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:10:51 (1787123451) [ 5071.842716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5071.847836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5071.855897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5072.354784] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5072.388825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5072.402272] Lustre: Skipped 2 previous similar messages [ 5076.110497] Lustre: server umount lustre-MDT0000 complete [ 5077.472573] LustreError: 160364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5077.495848] LustreError: 160364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5080.395424] LustreError: 159543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123709 with bad export cookie 4339277740631829605 [ 5080.402580] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5080.406535] LustreError: 159543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5082.591866] LustreError: 160779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5082.592420] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5082.649763] Lustre: Skipped 1 previous similar message [ 5082.660842] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5085.602260] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5087.315168] Lustre: server umount lustre-MDT0001 complete [ 5097.753669] Lustre: server umount lustre-OST0000 complete [ 5108.816399] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 5121.175251] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5162.857790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5163.102231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5163.129488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5163.234651] Lustre: lustre-MDT0000: new disk, initializing [ 5163.313252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5163.330586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5168.396285] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5180.628124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5180.810299] Lustre: 166185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5180.889296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5180.892748] Lustre: Skipped 1 previous similar message [ 5180.982615] Lustre: lustre-MDT0001: new disk, initializing [ 5181.134783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5181.243312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5181.264631] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5187.467449] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5199.747704] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5207.945343] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5217.036744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5217.310231] Lustre: lustre-OST0000: new disk, initializing [ 5217.313953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5217.324527] Lustre: 168155:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5217.419242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5223.562247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5223.568461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5223.704791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5224.903921] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5234.852951] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5240.795870] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5241.121211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5246.768624] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5247.087664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5249.505188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5249.509110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5249.515900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5249.527136] Lustre: Skipped 1 previous similar message [ 5249.542954] Lustre: Skipped 1 previous similar message [ 5255.427194] Lustre: server umount lustre-OST0000 complete [ 5259.746510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5259.748582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5259.765726] Lustre: Skipped 1 previous similar message [ 5259.780456] Lustre: Skipped 3 previous similar messages [ 5265.668940] Lustre: server umount lustre-MDT0000 complete [ 5269.567894] LustreError: 168165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123898 with bad export cookie 4339277740631988435 [ 5269.581400] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5269.583723] LustreError: 168165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5269.898930] Lustre: server umount lustre-MDT0001 complete [ 5279.004621] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:18:25 (1787123905) [ 5295.445627] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 5306.038200] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5344.568053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5344.909966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5344.936979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5345.002764] Lustre: lustre-MDT0000: new disk, initializing [ 5345.072121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5345.103671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5349.665684] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5360.172359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5360.340759] Lustre: 172912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5360.573738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5365.627600] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5376.972538] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5384.147139] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5391.401347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5391.791797] Lustre: lustre-OST0000: new disk, initializing [ 5391.800733] Lustre: Skipped 1 previous similar message [ 5391.808404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5391.815514] Lustre: Skipped 2 previous similar messages [ 5391.825779] Lustre: 174891:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5398.451601] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5400.248390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5400.269230] Lustre: Skipped 1 previous similar message [ 5400.276703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5400.401819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5407.664969] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5415.905916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5415.911192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5415.916779] LustreError: Skipped 2 previous similar messages [ 5415.917466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5415.917469] Lustre: Skipped 2 previous similar messages [ 5415.935959] Lustre: Skipped 1 previous similar message [ 5420.770635] Lustre: server umount lustre-OST0000 complete [ 5432.853284] Lustre: server umount lustre-MDT0000 complete [ 5435.365297] LustreError: 173917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5435.393519] LustreError: 173917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5436.862432] LustreError: 172904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124065 with bad export cookie 4339277740631989317 [ 5436.862972] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5436.867295] LustreError: 172904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5437.204911] Lustre: server umount lustre-MDT0001 complete [ 5449.181540] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:21:15 (1787124075) [ 5466.940747] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 5477.806781] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5516.431402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5516.700736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5516.727582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5516.821232] Lustre: lustre-MDT0000: new disk, initializing [ 5516.917028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5516.918977] Lustre: Skipped 2 previous similar messages [ 5516.931315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5521.819150] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5532.322637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5532.518650] Lustre: 179217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5532.934653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5537.401973] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5547.651863] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5554.058576] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5561.510357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5561.920415] Lustre: 181191:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5568.955157] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5569.618760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5569.811989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5579.791258] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5590.497325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5590.498908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5590.504361] Lustre: Skipped 3 previous similar messages [ 5590.505142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5590.505146] Lustre: Skipped 5 previous similar messages [ 5593.935526] Lustre: server umount lustre-OST0000 complete [ 5599.720541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5599.746193] LustreError: Skipped 1 previous similar message [ 5605.453382] Lustre: server umount lustre-MDT0000 complete [ 5605.862525] LustreError: 179228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5610.362045] LustreError: 179210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124239 with bad export cookie 4339277740631990402 [ 5610.373165] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5610.386993] LustreError: 179210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5611.188342] Lustre: server umount lustre-MDT0001 complete [ 5622.350603] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 5625.567388] Key type lgssc unregistered [ 5625.878621] LNet: 183051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5625.884539] LNetError: 183051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5625.908363] LNet: Removed LNI 192.168.202.151@tcp [ 5627.282095] Key type .llcrypt unregistered [ 5627.284825] Key type ._llcrypt unregistered [ 5655.202851] Key type ._llcrypt registered [ 5655.204938] Key type .llcrypt registered [ 5655.371171] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5657.035827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5657.171801] alg: No test for adler32 (adler32-zlib) [ 5658.841320] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 5659.198120] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 5660.896818] Key type lgssc registered [ 5662.531712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5671.423631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5671.450500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.192207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5678.333176] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5682.661822] LustreError: 184468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5682.685971] LustreError: 184468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5687.784943] LustreError: 184469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.936587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.280703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5693.714135] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5701.950098] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5708.637041] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5715.464173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5715.747463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5717.829551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5722.029739] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5729.411034] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5753.824926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5753.826615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.841031] LustreError: Skipped 1 previous similar message [ 5753.841884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5753.880614] Lustre: Skipped 1 previous similar message [ 5756.778213] Lustre: server umount lustre-OST0000 complete [ 5764.069062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5764.076916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5764.093604] Lustre: Skipped 1 previous similar message [ 5764.094812] Lustre: Skipped 1 previous similar message [ 5769.185916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5769.191521] Lustre: Skipped 2 previous similar messages [ 5773.281436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5775.327171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5775.796867] Lustre: server umount lustre-MDT0000 complete [ 5778.401858] LustreError: 184464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.416310] LustreError: 184464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5779.820877] LustreError: 184449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124408 with bad export cookie 2954046231422417369 [ 5779.833382] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5779.834140] LustreError: 184449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5780.371903] Lustre: server umount lustre-MDT0001 complete [ 5794.520895] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 5797.466428] Key type lgssc unregistered [ 5797.703696] LNet: 188030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5797.714450] LNetError: 188030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5797.747906] LNet: Removed LNI 192.168.202.151@tcp [ 5798.914905] Key type .llcrypt unregistered [ 5798.917788] Key type ._llcrypt unregistered [ 5824.694040] Key type ._llcrypt registered [ 5824.697060] Key type .llcrypt registered [ 5824.828903] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5826.003920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5826.035501] alg: No test for adler32 (adler32-zlib) [ 5827.287218] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 5827.472544] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 5829.111586] Key type lgssc registered [ 5830.331454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5839.223437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5839.258580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5841.051457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5846.625255] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5850.596346] LustreError: 189448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5850.617520] LustreError: 189448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5855.719653] LustreError: 189447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5857.196319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.649371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5862.788820] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5871.381447] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5877.898059] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5886.401851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5886.848025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5894.056383] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5903.377791] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5910.391663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5920.749245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5920.751897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5920.776717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5920.804370] Lustre: Skipped 1 previous similar message [ 5922.375126] Lustre: server umount lustre-OST0000 complete [ 5930.871976] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9a4f10318800.ost_server_uuid 50 [ 5944.323426] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5960.163409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5960.170121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.180915] Lustre: Skipped 1 previous similar message [ 5960.189499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5961.697458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5961.704573] Lustre: Skipped 2 previous similar messages [ 5966.159412] Lustre: server umount lustre-MDT0000 complete [ 5966.826142] LustreError: 190176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5966.847696] LustreError: 190176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5970.454357] LustreError: 189427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124599 with bad export cookie 6807436495769437787 [ 5970.466655] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5970.472511] LustreError: 189427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5971.058746] Lustre: server umount lustre-MDT0001 complete [ 5981.478648] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:30:07 (1787124607) [ 5989.605818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5990.279810] LustreError: 193409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5990.410042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5995.491533] LustreError: 193410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5995.870978] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6005.690331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.730098] LustreError: 193410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.757239] LustreError: 193410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6006.324037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6011.878869] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6020.705467] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6029.594278] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6039.501170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6039.772361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6046.796398] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6056.328787] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6060.313956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6064.413523] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6064.729983] Lustre: lustre-OST0001: new disk, initializing [ 6064.737411] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6064.742753] Lustre: 196253:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6064.948118] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6070.387163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6070.400422] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6070.567276] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6071.799986] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6082.765468] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6086.124099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6086.127973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6086.142922] Lustre: Skipped 2 previous similar messages [ 6086.148948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6091.268078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6091.271401] Lustre: Skipped 3 previous similar messages [ 6092.019475] Lustre: server umount lustre-OST0000 complete [ 6094.046750] LustreError: 195231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.591411] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6099.830769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6105.307176] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6105.591814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6121.958163] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6121.958382] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.964283] LustreError: Skipped 1 previous similar message [ 6121.966622] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6121.992529] Lustre: Skipped 1 previous similar message [ 6126.877542] Lustre: server umount lustre-OST0001 complete [ 6132.200754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6132.214826] Lustre: Skipped 1 previous similar message [ 6137.898287] Lustre: server umount lustre-MDT0000 complete [ 6141.410236] LustreError: 194138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6141.444143] LustreError: 194138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 6142.465349] LustreError: 193389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124771 with bad export cookie 6807436495769438858 [ 6142.480378] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6143.039626] Lustre: server umount lustre-MDT0001 complete [ 6172.201919] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:33:18 (1787124798) [ 6180.076532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.358946] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6180.395848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6180.571418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6185.646987] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6194.635114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6194.793935] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6194.898978] Lustre: 199694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6202.165174] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6212.200643] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6218.854763] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6226.421072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6226.762749] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6234.877791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6235.252486] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6243.387446] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6252.637067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6252.711980] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6253.101632] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6253.104366] Lustre: Skipped 2 previous similar messages [ 6259.298523] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6267.357045] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6270.434708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6270.453947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6270.475104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6270.478085] Lustre: Skipped 5 previous similar messages [ 6284.255202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6284.578318] Lustre: server umount lustre-OST0000 complete [ 6286.311501] LustreError: 201544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6286.349592] LustreError: 201544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6306.790440] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6306.798732] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6306.822905] Lustre: Skipped 1 previous similar message [ 6306.826781] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6306.826790] Lustre: Skipped 8 previous similar messages [ 6306.863777] LustreError: Skipped 2 previous similar messages [ 6308.567252] Lustre: server umount lustre-OST0001 complete [ 6318.849405] Lustre: server umount lustre-MDT0000 complete [ 6323.496178] LustreError: 201563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124952 with bad export cookie 6807436495769440265 [ 6323.507193] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6323.512152] LustreError: 201563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6323.860965] Lustre: server umount lustre-MDT0001 complete [ 6351.833217] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:36:18 (1787124978) [ 6367.254833] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 6376.668433] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 6410.104144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6410.381831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6410.416989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6410.512057] Lustre: lustre-MDT0000: new disk, initializing [ 6410.635206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6410.670057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6415.284152] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6426.273911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6426.458863] Lustre: 207802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6426.542599] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6426.552124] Lustre: Skipped 1 previous similar message [ 6426.597147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6426.607351] Lustre: Skipped 4 previous similar messages [ 6426.687615] Lustre: lustre-MDT0001: new disk, initializing [ 6426.835909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6426.853204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6432.693385] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6445.480217] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6452.657933] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6462.269312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6462.769860] Lustre: lustre-OST0000: new disk, initializing [ 6462.776949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6462.805405] Lustre: 209772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6469.543289] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6470.232979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6470.240198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6470.361532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6480.697846] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6486.754435] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6487.056289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6492.320072] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6492.560322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6496.229488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.259087] Lustre: Skipped 4 previous similar messages [ 6496.271340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6496.279519] LustreError: Skipped 1 previous similar message [ 6509.023696] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6509.311747] Lustre: server umount lustre-OST0000 complete [ 6520.011245] Lustre: server umount lustre-MDT0000 complete [ 6521.829920] LustreError: 207814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6521.859000] LustreError: 207814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 6524.219078] LustreError: 207794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125153 with bad export cookie 6807436495769441833 [ 6524.228977] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6524.237437] LustreError: 207794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6524.908058] Lustre: server umount lustre-MDT0001 complete [ 6534.935504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6542.186645] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6550.739444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6556.095703] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6564.473447] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6571.333857] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6581.454108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6591.438508] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6600.482270] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6605.154803] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6606.439788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6610.962314] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6611.216542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6616.038890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6616.041916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6616.071064] Lustre: Skipped 2 previous similar messages [ 6616.074044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6616.074051] Lustre: Skipped 9 previous similar messages [ 6616.114768] LustreError: Skipped 1 previous similar message [ 6619.973289] Lustre: server umount lustre-OST0000 complete [ 6628.968422] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6635.076471] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6666.221641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6666.736612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6666.746449] Lustre: Skipped 5 previous similar messages [ 6674.193081] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6682.097818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6682.119911] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6682.120671] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6682.161583] Lustre: Skipped 1 previous similar message [ 6683.310996] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6685.139909] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6692.328791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6692.338629] LustreError: Skipped 2 previous similar messages [ 6696.771512] Lustre: server umount lustre-OST0000 complete [ 6709.056196] Lustre: server umount lustre-MDT0000 complete [ 6713.775318] LustreError: 211973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125342 with bad export cookie 6807436495769442715 [ 6713.778751] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6713.797036] LustreError: 211973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6714.304827] Lustre: server umount lustre-MDT0001 complete [ 6724.291770] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:42:30 (1787125350) [ 6731.766813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6738.298597] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6747.539641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.819945] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6762.255690] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6769.470250] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6776.775558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6784.681454] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6792.156445] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6796.882569] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6797.176432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6802.530934] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6802.727767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6809.854689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6810.067337] Lustre: lustre-OST0001: new disk, initializing [ 6810.072052] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6810.080608] Lustre: 221339:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6818.557511] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6819.959935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6819.975596] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6820.148754] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6828.962479] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6834.448792] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6834.745683] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6839.391118] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6839.560119] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6845.958427] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6845.960368] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6845.990515] Lustre: Skipped 6 previous similar messages [ 6847.729665] Lustre: server umount lustre-OST0001 complete [ 6851.044774] LustreError: 219793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6851.076991] LustreError: 219793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6854.562925] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6854.786491] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6859.386553] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6859.678870] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6889.702714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6895.092244] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6895.104753] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6895.107551] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6897.345563] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 6905.856736] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6907.729825] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6910.439375] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6910.451298] Lustre: Skipped 10 previous similar messages [ 6916.391763] Lustre: server umount lustre-OST0001 complete [ 6941.736618] Lustre: server umount lustre-OST0000 complete [ 6953.061400] Lustre: server umount lustre-MDT0000 complete [ 6957.796766] LustreError: 217950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125586 with bad export cookie 6807436495769443681 [ 6957.817401] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6972.383418] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6972.900953] Lustre: server umount lustre-MDT0001 complete [ 7003.346943] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:47:09 (1787125629) [ 7012.233833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.580857] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7012.656352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7018.773609] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7029.427743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7029.570866] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7029.625587] Lustre: 226587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7035.676115] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7045.275362] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7052.566301] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7061.799387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7062.187053] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7071.007909] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7082.049562] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7093.121760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7093.295129] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7101.667872] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7111.794422] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7113.752904] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7120.175547] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7120.487857] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7125.835879] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7126.069701] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7132.914195] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a4f048f9800.ost_server_uuid 50 [ 7135.338380] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a4f048f9800.ost_server_uuid in IDLE state after 0 sec [ 7136.729184] Lustre: Permanently deactivating lustre-OST0001 [ 7136.772339] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7140.945138] Lustre: Permanently reactivating lustre-OST0001 [ 7141.008642] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7141.015740] Lustre: Skipped 2 previous similar messages [ 7149.541827] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7149.547464] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7149.559670] LustreError: Skipped 3 previous similar messages [ 7149.588621] Lustre: Skipped 7 previous similar messages [ 7153.783882] Lustre: server umount lustre-OST0001 complete [ 7164.639907] Lustre: server umount lustre-OST0000 complete [ 7174.837777] Lustre: server umount lustre-MDT0000 complete [ 7179.741730] LustreError: 226581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125808 with bad export cookie 6807436495769444969 [ 7179.766510] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7179.772272] LustreError: 226581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7180.789221] Lustre: server umount lustre-MDT0001 complete [ 7212.544938] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:50:38 (1787125838) [ 7222.077745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7222.559125] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7222.642933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7222.655276] Lustre: Skipped 2 previous similar messages [ 7223.050289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7223.057941] Lustre: Skipped 9 previous similar messages [ 7228.845183] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7238.580149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7238.789434] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7238.835538] Lustre: 233287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7245.175323] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7254.519934] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7260.619560] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7268.894474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7269.176284] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7269.222897] Lustre: Permanently deactivating lustre-OST0000 [ 7269.252231] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7276.890109] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7278.128579] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7285.759974] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7294.069135] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7294.223284] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7301.541345] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7302.317285] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7313.740217] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7315.576413] Lustre: Permanently reactivating lustre-OST0000 [ 7315.599685] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7315.608995] Lustre: Skipped 2 previous similar messages [ 7323.280993] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7323.291572] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 45 seconds. I think it's dead, and I am evicting it. exp 00000000fd01ead9, cur 1787125952 deadline 1787125937 last 1787125907 [ 7323.297024] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7323.343166] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7323.355992] Lustre: Skipped 1 previous similar message [ 7323.419442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7347.168089] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7347.488996] Lustre: server umount lustre-OST0001 complete [ 7357.783042] Lustre: server umount lustre-OST0000 complete [ 7367.946598] Lustre: server umount lustre-MDT0000 complete [ 7368.166366] LustreError: 233293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7368.198827] LustreError: 233293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 7371.730440] LustreError: 233279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126000 with bad export cookie 6807436495769446649 [ 7371.746396] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7371.749935] LustreError: 233279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7372.715755] Lustre: server umount lustre-MDT0001 complete [ 7390.969782] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7394.370174] Key type lgssc unregistered [ 7394.763345] LNet: 238197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7394.775785] LNetError: 238197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7394.796706] LNet: Removed LNI 192.168.202.151@tcp [ 7396.075667] Key type .llcrypt unregistered [ 7396.077817] Key type ._llcrypt unregistered [ 7420.283898] Key type ._llcrypt registered [ 7420.285263] Key type .llcrypt registered [ 7420.383397] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7421.933438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7422.319419] alg: No test for adler32 (adler32-zlib) [ 7423.512616] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 7423.763796] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7425.536156] Key type lgssc registered [ 7427.163458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7439.068649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7439.102349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7441.056879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7446.636343] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7450.595642] LustreError: 239664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7450.625741] LustreError: 239664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7455.714782] LustreError: 239665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7456.630939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7456.740975] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7456.783709] Lustre: 239651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7456.792314] Lustre: Permanently deactivating lustre-MDT0001 [ 7456.803296] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7457.092220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7462.398854] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7465.028545] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7471.134385] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7477.159177] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7485.654772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7485.860182] LustreError: 241505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7485.872865] LustreError: 241505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7486.138698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7491.576527] LustreError: 241507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7491.608293] LustreError: 241507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7491.626524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7493.397159] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7496.675413] LustreError: 241507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7501.836238] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7503.150370] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7506.938237] LustreError: 241507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7506.957986] LustreError: 241507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7511.878873] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7512.221770] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7517.720687] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7517.725680] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7521.595134] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7531.726156] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7536.178524] Lustre: Permanently reactivating lustre-MDT0001 [ 7536.207923] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7536.215483] Lustre: Skipped 2 previous similar messages [ 7540.171501] Lustre: lustre-MDT0001: Client 0c4bb61d-86aa-4964-ad5c-be036284262a (at 192.168.202.51@tcp) reconnecting [ 7543.053182] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.085535] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7543.106374] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7543.128879] LustreError: 243277:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7543.134277] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7543.143669] LustreError: 243277:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7543.143784] LustreError: 243277:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.151@tcp: cfg command failed: rc = -108 [ 7543.143803] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7543.143803] [ 7543.145506] LustreError: 239656:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7548.990255] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7550.926848] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7558.165746] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7558.380441] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7564.150312] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7564.407237] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7568.083098] Lustre: Permanently deactivating lustre-MDT0001 [ 7568.105530] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7568.110186] Lustre: Skipped 2 previous similar messages [ 7575.666481] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7589.344900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.350846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7589.377598] Lustre: Skipped 1 previous similar message [ 7589.399513] Lustre: Skipped 3 previous similar messages [ 7592.834636] Lustre: server umount lustre-MDT0000 complete [ 7594.466322] LustreError: 239664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7594.500386] LustreError: 239664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7597.377644] LustreError: 244152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126226 with bad export cookie 7322757144647278511 [ 7597.379548] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7597.387605] LustreError: 244152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7597.746438] Lustre: server umount lustre-MDT0001 complete [ 7604.038369] Lustre: server umount lustre-OST0000 complete [ 7617.243910] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:57:23 (1787126243) [ 7632.034884] Lustre: server umount lustre-OST0001 complete [ 7640.668888] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 7653.344830] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7699.823435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7700.091955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7700.099849] Lustre: Skipped 2 previous similar messages [ 7700.124494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7700.236098] Lustre: lustre-MDT0000: new disk, initializing [ 7700.360812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7700.384499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7706.565072] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7720.790926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7721.005986] Lustre: 247983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7721.088553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7721.094322] Lustre: Skipped 1 previous similar message [ 7721.261487] Lustre: lustre-MDT0001: new disk, initializing [ 7721.466610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7721.547876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7721.569560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7727.662350] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7739.633375] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7746.781255] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7755.106185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7755.581392] Lustre: lustre-OST0000: new disk, initializing [ 7755.584860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7755.593575] Lustre: 249950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7755.692778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7763.102878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7763.128450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7763.288131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7763.755764] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7774.662435] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7777.218200] LustreError: 247987:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7787.224069] LustreError: 247987:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7787.248184] LustreError: 250913:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7787.397162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7787.629825] Lustre: lustre-OST0001: new disk, initializing [ 7787.636208] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7787.643301] Lustre: 251066:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7787.801938] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7795.482880] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7796.284679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7796.297284] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7796.405760] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7797.295269] LustreError: 250913:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7806.996906] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7809.041945] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7812.064532] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7812.071946] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7812.093343] Lustre: Skipped 2 previous similar messages [ 7812.106510] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7812.113244] Lustre: Skipped 1 previous similar message [ 7817.211114] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7817.226350] Lustre: Skipped 1 previous similar message [ 7817.669516] Lustre: server umount lustre-OST0001 complete [ 7822.322576] LustreError: 251922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7822.338695] LustreError: 251922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7837.668245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7837.684504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7837.694124] Lustre: Skipped 1 previous similar message [ 7837.714755] Lustre: Skipped 1 previous similar message [ 7842.890779] Lustre: server umount lustre-OST0000 complete [ 7847.919479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7847.925317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7847.931673] Lustre: Skipped 1 previous similar message [ 7847.949341] Lustre: Skipped 1 previous similar message [ 7854.015688] Lustre: server umount lustre-MDT0000 complete [ 7859.140951] LustreError: 247973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126488 with bad export cookie 7322757144647281542 [ 7859.153489] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7859.156184] LustreError: 247973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7865.848326] Lustre: server umount lustre-MDT0001 complete [ 7879.730192] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7883.307613] Key type lgssc unregistered [ 7883.677584] LNet: 253113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7883.682076] LNetError: 253113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7883.693123] LNet: Removed LNI 192.168.202.151@tcp [ 7884.682120] Key type .llcrypt unregistered [ 7884.686325] Key type ._llcrypt unregistered [ 7937.579991] Key type ._llcrypt registered [ 7937.580901] Key type .llcrypt registered [ 7937.703207] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7939.612499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7939.649581] alg: No test for adler32 (adler32-zlib) [ 7940.714943] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 7941.135249] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7942.976973] Key type lgssc registered [ 7944.178663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7954.242807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7954.254528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7955.536486] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7955.568327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7955.822554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7962.786440] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7975.262152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7975.369837] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7975.417520] Lustre: 255322:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7975.915025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7983.965440] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7995.032925] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8002.061198] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8010.099847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8010.291891] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8010.621080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8017.137624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8018.436990] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8028.092309] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8093.668937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8093.692708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8093.725719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8094.185785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8094.193083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8096.946376] Lustre: server umount lustre-OST0000 complete [ 8102.155375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8117.593503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8118.077414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8120.125104] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8120.140748] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8125.027492] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8133.122899] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8134.530990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8134.560192] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8175.585454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8175.590810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8175.595906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8175.601018] Lustre: Skipped 1 previous similar message [ 8175.614250] Lustre: Skipped 2 previous similar messages [ 8179.375963] Lustre: server umount lustre-OST0000 complete [ 8185.825316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8185.827927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8185.843635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8185.859726] Lustre: Skipped 1 previous similar message [ 8190.228315] LustreError: 260002:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000036ba7417) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 8192.376633] Lustre: server umount lustre-MDT0000 complete [ 8196.085983] LustreError: 255335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8196.107286] LustreError: 255335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8196.753845] LustreError: 257191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126825 with bad export cookie 1562906992284812541 [ 8196.754674] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8196.786450] LustreError: 257191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8197.287818] Lustre: server umount lustre-MDT0001 complete [ 8210.180577] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8213.809891] Key type lgssc unregistered [ 8214.160974] LNet: 260755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8214.173534] LNetError: 260755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8214.208527] LNet: Removed LNI 192.168.202.151@tcp [ 8215.399797] Key type .llcrypt unregistered [ 8215.403358] Key type ._llcrypt unregistered [ 8243.370359] Key type ._llcrypt registered [ 8243.372828] Key type .llcrypt registered [ 8243.525392] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8245.297967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8245.327607] alg: No test for adler32 (adler32-zlib) [ 8246.506600] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 8246.771760] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8248.495502] Key type lgssc registered [ 8249.931934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8257.964899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8258.007346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8259.478503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8265.153797] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8269.799792] LustreError: 262174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8269.840669] LustreError: 262174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8274.913952] LustreError: 262175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8275.547949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8275.866787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8282.321526] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8292.512859] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8300.383517] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8308.825938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8309.382632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8316.748439] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8326.272671] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8329.846333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8360.932450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8360.936175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8360.953501] Lustre: Skipped 1 previous similar message [ 8360.967912] Lustre: Skipped 1 previous similar message [ 8364.179436] Lustre: server umount lustre-OST0000 complete [ 8371.169401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8371.172906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8371.192911] Lustre: Skipped 1 previous similar message [ 8371.210641] Lustre: Skipped 1 previous similar message [ 8374.919728] Lustre: server umount lustre-MDT0000 complete [ 8376.302816] LustreError: 262174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8376.325352] LustreError: 262174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8379.361674] LustreError: 262155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127008 with bad export cookie 6041997215960019748 [ 8379.378755] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8379.386980] LustreError: 262155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8380.158100] Lustre: server umount lustre-MDT0001 complete [ 8393.685233] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8397.061555] Key type lgssc unregistered [ 8397.512762] LNet: 266487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8397.519399] LNetError: 266487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8397.560283] LNet: Removed LNI 192.168.202.151@tcp [ 8398.752900] Key type .llcrypt unregistered [ 8398.756113] Key type ._llcrypt unregistered [ 8417.711666] Key type ._llcrypt registered [ 8417.713528] Key type .llcrypt registered [ 8417.805227] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8418.508651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8418.573327] alg: No test for adler32 (adler32-zlib) [ 8419.733547] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 8419.978232] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8421.663194] Key type lgssc registered [ 8422.914758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8430.522968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8430.561024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8432.023202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8437.502538] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8441.314702] LustreError: 267709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8441.341925] LustreError: 267709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8446.437309] LustreError: 267710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8447.266101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8447.712389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8453.541579] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8462.725211] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8469.215726] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8476.331767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8476.589113] LustreError: 269531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8476.855548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8482.319120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8484.685041] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8493.229739] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8507.683915] Lustre: server umount lustre-OST0000 complete [ 8512.993922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8512.997263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8513.013165] Lustre: Skipped 1 previous similar message [ 8513.026936] Lustre: Skipped 1 previous similar message [ 8518.113616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8518.126271] Lustre: Skipped 1 previous similar message [ 8523.103744] Lustre: 267021:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787127136/real 1787127136] req@00000000f1d14045 x1873938531513728/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787127152 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8523.136535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8523.162326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8523.175329] Lustre: Skipped 1 previous similar message [ 8524.191156] Lustre: 267024:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787127136/real 1787127136] req@00000000c792041c x1873938531513984/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787127152 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8526.303894] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8526.592047] Lustre: server umount lustre-MDT0000 complete [ 8528.159317] Lustre: 267023:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787127141/real 1787127141] req@00000000ea1a7167 x1873938531515776/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787127157 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8528.354816] LustreError: 268867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8528.387548] LustreError: 268867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8530.484640] LustreError: 267689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127159 with bad export cookie 396379795495856360 [ 8530.491963] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8530.498139] LustreError: 267689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8531.019515] Lustre: server umount lustre-MDT0001 complete [ 8542.922865] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8546.058070] Key type lgssc unregistered [ 8546.378732] LNet: 271390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8546.389616] LNetError: 271390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8546.409330] LNet: Removed LNI 192.168.202.151@tcp [ 8547.470965] Key type .llcrypt unregistered [ 8547.473080] Key type ._llcrypt unregistered [ 8577.473694] Key type ._llcrypt registered [ 8577.476666] Key type .llcrypt registered [ 8577.795307] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8579.594236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8579.652235] alg: No test for adler32 (adler32-zlib) [ 8581.074968] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 8581.604689] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8583.495169] Key type lgssc registered [ 8585.498450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8595.901720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8595.943889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8597.428630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8601.747441] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8607.723352] LustreError: 272809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8607.737930] LustreError: 272809:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8610.528409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8610.851853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8615.448934] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8626.542070] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8634.201313] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8642.496924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8643.177620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8650.479912] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8658.503318] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8667.763096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8693.737309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8693.742215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8693.756347] Lustre: Skipped 1 previous similar message [ 8693.756746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8693.775023] Lustre: Skipped 1 previous similar message [ 8698.859098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8698.864549] Lustre: Skipped 1 previous similar message [ 8699.597804] Lustre: server umount lustre-OST0000 complete [ 8703.969161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8703.976255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8703.983888] Lustre: Skipped 1 previous similar message [ 8704.002247] Lustre: Skipped 1 previous similar message [ 8709.105161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8709.107696] Lustre: Skipped 1 previous similar message [ 8709.757562] Lustre: server umount lustre-MDT0000 complete [ 8713.437141] LustreError: 272789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127342 with bad export cookie 16206962479445668591 [ 8713.447739] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8713.723880] Lustre: server umount lustre-MDT0001 complete [ 8724.932551] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8727.878755] Key type lgssc unregistered [ 8728.153710] LNet: 277140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8728.164169] LNetError: 277140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8729.198328] LNet: Removed LNI 192.168.202.151@tcp [ 8730.172183] Key type .llcrypt unregistered [ 8730.174104] Key type ._llcrypt unregistered [ 8748.190948] Key type ._llcrypt registered [ 8748.192403] Key type .llcrypt registered [ 8748.313507] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8749.460763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8749.641586] alg: No test for adler32 (adler32-zlib) [ 8750.769468] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 8750.979773] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8752.615350] Key type lgssc registered [ 8753.611939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8761.685216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8761.717687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8763.540667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8768.133554] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8774.112834] LustreError: 278376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8774.140799] LustreError: 278376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8777.547789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8777.941139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8782.827519] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8791.086769] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8797.473418] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8804.944822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8805.428334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.550454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8812.328992] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8821.913572] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8835.049296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8835.061055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.095499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8835.651983] Lustre: server umount lustre-OST0000 complete [ 8843.238873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8843.257055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8843.260612] Lustre: Skipped 1 previous similar message [ 8846.902976] Lustre: server umount lustre-MDT0000 complete [ 8848.361219] LustreError: 278386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8848.382188] LustreError: 278386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8851.135639] LustreError: 278342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127480 with bad export cookie 18364418043568202880 [ 8851.141983] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8851.146752] LustreError: 278342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8862.094642] Lustre: server umount lustre-MDT0001 complete [ 8876.644361] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8880.251913] Key type lgssc unregistered [ 8880.545610] LNet: 282077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8880.549692] LNetError: 282077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8880.559763] LNet: Removed LNI 192.168.202.151@tcp [ 8881.692612] Key type .llcrypt unregistered [ 8881.694305] Key type ._llcrypt unregistered [ 8894.877444] Key type ._llcrypt registered [ 8894.882974] Key type .llcrypt registered [ 8895.183956] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8913.307527] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 8928.944582] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8930.387300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8930.428835] alg: No test for adler32 (adler32-zlib) [ 8931.670697] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 8931.946158] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8933.663146] Key type lgssc registered [ 8934.903266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8972.669923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8972.696782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8973.980893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8974.012073] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8974.086952] Lustre: lustre-MDT0000: new disk, initializing [ 8974.177158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8974.222141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8979.652917] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8989.853856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8990.068364] Lustre: 285634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8990.125873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8990.129672] Lustre: Skipped 1 previous similar message [ 8990.215157] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8990.218513] Lustre: lustre-MDT0001: new disk, initializing [ 8990.312858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8990.345367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8990.358541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8995.708600] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9005.168167] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9010.360129] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9017.054816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9017.276204] Lustre: lustre-OST0000: new disk, initializing [ 9017.278790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9017.283357] Lustre: 287603:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9017.365199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9024.354620] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9027.195964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9027.207964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9027.395731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9033.715324] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9039.033898] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9039.217900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9044.235469] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9044.505426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9048.036463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9048.042984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9048.050113] LustreError: Skipped 1 previous similar message [ 9048.054497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9048.091306] Lustre: Skipped 1 previous similar message [ 9052.548186] Lustre: server umount lustre-OST0000 complete [ 9058.274104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9058.293355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9058.304988] Lustre: Skipped 2 previous similar messages [ 9063.750098] Lustre: server umount lustre-MDT0000 complete [ 9067.843486] LustreError: 285624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127696 with bad export cookie 11796250551849218312 [ 9067.844755] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9067.850627] LustreError: 285624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9068.211199] Lustre: server umount lustre-MDT0001 complete [ 9077.930190] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 04:21:44 (1787127704) [ 9084.892423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9085.450872] LustreError: 290015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9085.488269] LustreError: 290015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9085.582899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9090.536689] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9091.050139] LustreError: 290016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9095.143235] LustreError: 290015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9098.988629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9099.472989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9104.250339] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9111.111840] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9116.674827] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9122.775790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9123.218024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9129.281294] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9135.711913] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9145.827363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9145.831167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9145.839468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9145.860247] Lustre: Skipped 2 previous similar messages [ 9145.879553] Lustre: Skipped 1 previous similar message [ 9150.955807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9150.962462] Lustre: Skipped 1 previous similar message [ 9151.138771] Lustre: server umount lustre-OST0000 complete [ 9156.065059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9156.070927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9156.080244] Lustre: Skipped 1 previous similar message [ 9161.186769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9161.199784] Lustre: Skipped 3 previous similar messages [ 9169.376411] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9169.525262] Lustre: server umount lustre-MDT0000 complete [ 9171.425697] LustreError: 290015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9171.440236] LustreError: 290015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9173.954819] LustreError: 289996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127802 with bad export cookie 11796250551849219187 [ 9173.988568] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9174.551343] Lustre: server umount lustre-MDT0001 complete [ 9188.621612] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 9192.681768] Key type lgssc unregistered [ 9193.242183] LNet: 293552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9193.253121] LNetError: 293552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9193.280453] LNet: Removed LNI 192.168.202.151@tcp [ 9194.543298] Key type .llcrypt unregistered [ 9194.545256] Key type ._llcrypt unregistered [ 9223.305456] Key type ._llcrypt registered [ 9223.307471] Key type .llcrypt registered [ 9223.406607] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 9239.811315] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9241.495495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9241.750530] alg: No test for adler32 (adler32-zlib) [ 9243.224132] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 9243.546055] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9245.487296] Key type lgssc registered [ 9247.376035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9296.807825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9296.839556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.097982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9298.115933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9298.209499] Lustre: lustre-MDT0000: new disk, initializing [ 9298.321520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9298.344181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9302.831681] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9312.979387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9313.058755] Lustre: 297383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9313.110196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9313.112983] Lustre: Skipped 1 previous similar message [ 9313.127137] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9313.194446] Lustre: lustre-MDT0001: new disk, initializing [ 9313.296636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9313.350678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9313.359595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9318.692896] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9328.986592] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9334.496349] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9341.373485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9341.726659] Lustre: lustre-OST2710: new disk, initializing [ 9341.731467] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9341.737206] Lustre: 299352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9341.835392] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9348.048331] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9350.712049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9350.715940] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9350.848681] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9357.253657] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9364.063263] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9364.213493] Lustre: lustre-OST03e8: new disk, initializing [ 9364.216604] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9364.221818] Lustre: 300416:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9364.326835] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9370.631490] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9370.679671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9370.684229] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9370.753077] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9377.415149] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9385.760719] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9386.066340] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9391.212416] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9391.556703] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9396.735577] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9396.928137] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9403.056660] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9403.409418] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9410.366674] Lustre: server umount lustre-MDT0000 complete [ 9410.534347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9410.541902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9411.583668] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9413.926858] LustreError: 297374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128042 with bad export cookie 10150180026390214137 [ 9413.959247] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9414.567801] Lustre: server umount lustre-MDT0001 complete [ 9429.055618] Lustre: server umount lustre-OST2710 complete [ 9442.825658] Lustre: server umount lustre-OST03e8 complete [ 9459.971122] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 9469.163875] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9502.905226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9503.157844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9503.185413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9503.300936] Lustre: lustre-MDT0000: new disk, initializing [ 9503.452976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9503.469619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9508.286254] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9519.811379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9519.934325] Lustre: 305667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9519.986110] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9519.989798] Lustre: Skipped 1 previous similar message [ 9520.078991] Lustre: lustre-MDT0001: new disk, initializing [ 9520.170212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9520.225415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9520.236190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9526.074742] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9537.841259] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9544.126660] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9550.737193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9551.106097] Lustre: lustre-OST0000: new disk, initializing [ 9551.108788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9551.117296] Lustre: 307635:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9551.196977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9557.077890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9557.088609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9557.223531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9557.806052] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9567.531543] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9573.432047] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9573.808747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9579.505841] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9579.818519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9583.077811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9583.079388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9583.112159] Lustre: Skipped 2 previous similar messages [ 9583.120325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9588.204048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9588.209463] Lustre: Skipped 2 previous similar messages [ 9588.973493] Lustre: server umount lustre-OST0000 complete [ 9596.901288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9596.914708] LustreError: Skipped 1 previous similar message [ 9596.923253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9596.945140] Lustre: Skipped 1 previous similar message [ 9596.962905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9600.562448] Lustre: server umount lustre-MDT0000 complete [ 9603.553185] LustreError: 306476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9603.581924] LustreError: 306476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9604.856962] LustreError: 305658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128233 with bad export cookie 10150180026390215817 [ 9604.861331] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9604.881528] LustreError: 305658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9605.590820] Lustre: server umount lustre-MDT0001 complete [ 9616.246351] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:30:42 (1787128242) [ 9617.812798] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9619.956494] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:30:45 (1787128245) [ 9623.044953] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9650.666992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9650.981726] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9651.001552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9651.176914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9656.133683] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9665.602376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9665.753703] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9665.804655] Lustre: 311496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9670.707327] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9692.737943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9693.067599] Lustre: Denying initial registration attempt from nid 192.168.202.151@tcp, specified as failover [ 9693.080779] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9693.096311] LustreError: 312951:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9693.102375] LustreError: 312951:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9693.105895] LustreError: 312951:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9693.219841] Lustre: server umount lustre-OST0000 complete [ 9693.224865] LustreError: 312951:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9702.368193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9702.376713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9702.410697] Lustre: Skipped 1 previous similar message [ 9702.423151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9702.431583] Lustre: Skipped 2 previous similar messages [ 9708.106250] Lustre: server umount lustre-MDT0000 complete [ 9710.561693] LustreError: 311509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9710.590147] LustreError: 311509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9713.120225] LustreError: 311488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128342 with bad export cookie 10150180026390216699 [ 9713.134317] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9713.135502] LustreError: 311488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9713.619356] Lustre: server umount lustre-MDT0001 complete [ 9728.302813] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 9732.232282] Key type lgssc unregistered [ 9732.521461] LNet: 314076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9732.546788] LNetError: 314076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9732.571267] LNet: Removed LNI 192.168.202.151@tcp [ 9733.681187] Key type .llcrypt unregistered [ 9733.683247] Key type ._llcrypt unregistered [ 9747.103405] Key type ._llcrypt registered [ 9747.105084] Key type .llcrypt registered [ 9747.266706] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9748.501185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9748.596821] alg: No test for adler32 (adler32-zlib) [ 9749.747872] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 9750.010036] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9751.695947] Key type lgssc registered [ 9753.012879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9806.340621] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9816.569268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9816.596406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9817.968446] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9818.010751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9818.196240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9823.344623] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9833.652087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9833.748777] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9833.779587] Lustre: 316840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9834.202388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9840.498185] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9850.061784] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9856.292814] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9877.426431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9877.805622] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9878.192266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9886.870714] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9896.249532] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9902.049319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9902.068741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9902.073443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9902.086494] Lustre: Skipped 1 previous similar message [ 9902.102791] Lustre: Skipped 1 previous similar message [ 9905.190348] Lustre: server umount lustre-OST0000 complete [ 9910.754882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9910.762902] LustreError: Skipped 1 previous similar message [ 9910.774754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9910.794020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9912.292305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9912.292982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9912.326068] Lustre: Skipped 1 previous similar message [ 9916.438057] Lustre: server umount lustre-MDT0000 complete [ 9917.414549] LustreError: 317594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9917.432784] LustreError: 317594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9920.856820] LustreError: 316834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128549 with bad export cookie 10722637305147958211 [ 9920.864425] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9920.865215] LustreError: 316834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9921.309738] Lustre: server umount lustre-MDT0001 complete [ 9936.327837] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 9940.294588] Key type lgssc unregistered [ 9940.755980] LNet: 320424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9940.770853] LNetError: 320424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9940.794255] LNet: Removed LNI 192.168.202.151@tcp [ 9941.707812] Key type .llcrypt unregistered [ 9941.710152] Key type ._llcrypt unregistered [ 9972.366791] Key type ._llcrypt registered [ 9972.368809] Key type .llcrypt registered [ 9972.534385] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9974.382650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9974.546889] alg: No test for adler32 (adler32-zlib) [ 9976.000785] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 9976.602740] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9978.423223] Key type lgssc registered [ 9980.182198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9990.744843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9990.771734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9992.737640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9999.189449] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10002.919993] LustreError: 321845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10002.949096] LustreError: 321845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10008.041801] LustreError: 321843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10012.687618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10013.448216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10019.840792] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10028.271636] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10034.883212] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10043.368233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10043.855133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10050.378994] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10058.377532] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10074.592993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10074.602299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10074.618412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10075.391184] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [10077.665951] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10077.666687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10077.684554] Lustre: Skipped 1 previous similar message [10077.704711] Lustre: Skipped 2 previous similar messages [10079.347299] Lustre: server umount lustre-MDT0000 complete [10080.482370] LustreError: 321838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10082.785500] LustreError: 321840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10082.815100] LustreError: 321840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10083.496677] LustreError: 321825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128712 with bad export cookie 7004131048161087443 [10083.510823] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10083.515515] LustreError: 321825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10085.614832] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [10087.907819] LustreError: 321840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10087.908277] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10087.954125] LustreError: 321840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10090.024957] Lustre: server umount lustre-MDT0001 complete [10097.732520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10106.486378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10108.735807] LustreError: 323678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10108.755492] LustreError: 323678:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10108.811209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10108.867896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10113.823338] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10119.104952] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10123.101077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10123.643461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10128.883634] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10129.013887] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10138.247693] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10140.006444] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10146.447038] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10147.902920] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10154.477922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10154.482124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10154.488526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10154.488536] Lustre: Skipped 2 previous similar messages [10164.714238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10164.721162] Lustre: Skipped 4 previous similar messages [10167.263513] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10167.412800] Lustre: server umount lustre-OST0000 complete [10174.944707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10174.949428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10174.987364] Lustre: Skipped 2 previous similar messages [10177.830308] Lustre: server umount lustre-MDT0000 complete [10180.064726] LustreError: 325467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10180.086230] LustreError: 325467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10181.973249] LustreError: 321825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128810 with bad export cookie 7004131048161094989 [10181.973858] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10181.987374] LustreError: 321825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10182.535545] Lustre: server umount lustre-MDT0001 complete [10196.172440] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [10199.513926] Key type lgssc unregistered [10199.892775] LNet: 328141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10199.904387] LNetError: 328141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10199.938255] LNet: Removed LNI 192.168.202.151@tcp [10201.087924] Key type .llcrypt unregistered [10201.090672] Key type ._llcrypt unregistered [10226.788274] Key type ._llcrypt registered [10226.791478] Key type .llcrypt registered [10226.970451] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10228.662326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10228.745216] alg: No test for adler32 (adler32-zlib) [10230.061766] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [10230.404040] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [10232.160274] Key type lgssc registered [10233.822748] Lustre: Echo OBD driver; http://www.lustre.org/ [10243.452248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10243.466240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10245.018612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10250.472404] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10254.311529] LustreError: 329557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10254.330908] LustreError: 329557:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10259.430849] LustreError: 329558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10260.142317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10260.809583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10266.160967] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10274.983441] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10281.646234] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10285.542099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10285.543295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10285.563333] Lustre: Skipped 1 previous similar message [10286.633320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10286.635963] Lustre: Skipped 1 previous similar message [10291.683145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10291.698383] Lustre: Skipped 1 previous similar message [10296.800733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10299.363347] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10299.639203] Lustre: server umount lustre-MDT0000 complete [10301.920465] LustreError: 330537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10301.926704] LustreError: 330537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10303.289729] LustreError: 329538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128932 with bad export cookie 16897285869267382876 [10303.293147] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10303.301866] LustreError: 329538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10303.639935] Lustre: server umount lustre-MDT0001 complete [10323.492024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10324.096612] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10324.130468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10324.353393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10329.570819] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10338.664956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10338.709715] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10338.722407] Lustre: 332147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10338.898154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10343.911605] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10353.301685] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10359.731695] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10370.992374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10371.343722] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10371.354726] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10371.371374] LustreError: 334029:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10371.380177] LustreError: 334029:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10371.390550] LustreError: 334029:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10371.513234] Lustre: server umount lustre-OST0000 complete [10371.516879] LustreError: 334029:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10383.479192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10383.806816] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10383.984203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10391.223425] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10392.702226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10399.654166] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10407.612583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10407.754393] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10407.828607] Lustre: lustre-OST0001: new disk, initializing [10407.832783] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10407.838482] Lustre: 335573:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10407.931055] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10415.070551] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10415.278553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10415.300120] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10415.577523] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10424.008416] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10430.953664] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10430.963075] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10430.973922] LustreError: Skipped 1 previous similar message [10431.021320] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10431.028759] Lustre: Skipped 2 previous similar messages [10441.202546] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10441.213063] Lustre: Skipped 3 previous similar messages [10442.207921] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10442.380974] Lustre: server umount lustre-OST0001 complete [10446.309823] LustreError: 335583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10446.354762] LustreError: 335583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10448.866281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10448.881512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10448.900586] Lustre: Skipped 1 previous similar message [10451.434395] LustreError: 334497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10451.453494] LustreError: 334497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10453.552935] Lustre: server umount lustre-OST0000 complete [10461.665261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10461.693919] Lustre: Skipped 1 previous similar message [10461.703802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10461.707027] Lustre: Skipped 4 previous similar messages [10464.904036] Lustre: server umount lustre-MDT0000 complete [10466.785451] LustreError: 336553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10466.816189] LustreError: 336553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10468.985640] LustreError: 334512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129097 with bad export cookie 16897285869267383429 [10468.986983] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10468.990538] LustreError: 334512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10469.369147] Lustre: server umount lustre-MDT0001 complete [10482.684760] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [10487.897345] Key type lgssc unregistered [10488.415418] LNet: 337541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10488.430557] LNetError: 337541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10489.514071] LNet: Removed LNI 192.168.202.151@tcp [10491.014130] Key type .llcrypt unregistered [10491.015597] Key type ._llcrypt unregistered [10575.395742] Key type ._llcrypt registered [10575.396788] Key type .llcrypt registered [10575.591308] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [10592.632812] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10594.066868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10594.087336] alg: No test for adler32 (adler32-zlib) [10595.190148] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [10595.472763] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [10597.151296] Key type lgssc registered [10598.835971] Lustre: Echo OBD driver; http://www.lustre.org/ [10637.377337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10637.445099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10638.841075] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10638.878114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10639.047677] Lustre: lustre-MDT0000: new disk, initializing [10639.206084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10639.241341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10646.828337] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10659.423367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10659.617450] Lustre: 342478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10659.711259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10659.724456] Lustre: Skipped 1 previous similar message [10659.835945] Lustre: lustre-MDT0001: new disk, initializing [10659.979590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10660.048845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10660.074313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10665.726324] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10676.669570] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10682.103445] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10691.684281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10692.237292] Lustre: lustre-OST0000: new disk, initializing [10692.246263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10692.267774] Lustre: 344448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10692.478755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10699.849168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10699.872201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10700.023291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10701.056611] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10712.218718] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10717.324879] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10717.518465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10723.129493] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10723.334722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10725.861166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10725.864896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10725.872583] LustreError: Skipped 1 previous similar message [10725.881875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10725.891135] Lustre: Skipped 1 previous similar message [10725.909863] Lustre: Skipped 1 previous similar message [10730.985293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10730.993191] Lustre: Skipped 1 previous similar message [10731.634510] Lustre: server umount lustre-OST0000 complete [10736.106235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10736.113613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10736.124579] Lustre: Skipped 1 previous similar message [10736.143956] Lustre: Skipped 1 previous similar message [10741.216692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10741.224139] Lustre: Skipped 1 previous similar message [10742.644989] Lustre: server umount lustre-MDT0000 complete [10746.940635] LustreError: 344447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129375 with bad export cookie 11536332676081862327 [10746.958350] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10746.964609] LustreError: 344447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10747.293734] Lustre: server umount lustre-MDT0001 complete [10757.532038] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:49:43 (1787129383) [10765.662355] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:49:51 (1787129391) [10775.332135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10775.749800] LustreError: 347206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10775.766581] LustreError: 347206:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10775.812471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10780.949745] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10781.167527] LustreError: 347207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10785.256318] LustreError: 347206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10790.372665] LustreError: 347207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10792.510459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10793.185744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10799.734845] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10807.680045] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10815.509475] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10823.706867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10824.139290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10832.282193] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10841.440859] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10855.071950] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10857.545710] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10859.614258] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10861.953027] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10864.076623] Lustre: Failing over lustre-MDT0000 [10864.275978] Lustre: server umount lustre-MDT0000 complete [10864.612560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10864.630424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10864.659075] LustreError: 348130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10873.035794] LustreError: 347203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10873.077153] LustreError: 347203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10882.529946] Lustre: 340704:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787129495/real 1787129495] req@000000009ca91e35 x1873940813261568/t0(0) o400->MGC192.168.202.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787129511 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10882.557856] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10884.930529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10892.776212] LustreError: 349698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10892.794405] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa019497260974e3c [10892.807441] LustreError: 349698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [10892.863726] Lustre: MGC192.168.202.151@tcp: Connection restored to 0@lo (at 0@lo) [10893.271459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10893.356428] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10893.525423] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10898.381715] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10898.413777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10898.467302] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10898.528254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10910.540967] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10912.729576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10916.807518] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10919.068369] Lustre: Failing over lustre-MDT0000 [10919.206652] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [10919.525556] Lustre: server umount lustre-MDT0000 complete [10924.002567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10924.017128] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10924.065440] Lustre: Skipped 3 previous similar messages [10929.122529] LustreError: 347207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10929.144422] LustreError: 347207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [10940.262079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10940.398605] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10940.777396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10940.860910] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10945.760311] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10946.043870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10946.047939] Lustre: Skipped 2 previous similar messages [10946.080849] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10946.124267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10946.373697] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10957.445169] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10959.436986] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10966.498230] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10966.519401] Lustre: Skipped 1 previous similar message [10966.523069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10966.529530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10972.780916] Lustre: server umount lustre-MDT0000 complete [10977.791559] LustreError: 347186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129606 with bad export cookie 11536332676081865043 [10977.801845] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10977.806241] LustreError: 347186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10978.187801] Lustre: server umount lustre-MDT0001 complete [10992.634279] Lustre: server umount lustre-OST0000 complete [11013.566070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11014.220494] LustreError: 353572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11014.232105] LustreError: 353572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [11014.306433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11019.998713] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11031.467251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11038.422516] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11048.527609] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11055.698756] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11065.751565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11074.922653] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11084.838919] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11085.751592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11101.153040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11101.166870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11101.170485] Lustre: Skipped 1 previous similar message [11101.181940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11101.188026] Lustre: Skipped 6 previous similar messages [11102.008416] Lustre: server umount lustre-OST0000 complete [11113.726191] Lustre: server umount lustre-MDT0000 complete [11118.495456] LustreError: 353552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129747 with bad export cookie 11536332676081865785 [11118.503274] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11118.515400] LustreError: 353552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11118.849710] Lustre: server umount lustre-MDT0001 complete [11132.430838] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [11136.685502] Key type lgssc unregistered [11137.074185] LNet: 357132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11137.082883] LNetError: 357132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11137.127273] LNet: Removed LNI 192.168.202.151@tcp [11138.000252] Key type .llcrypt unregistered [11138.001745] Key type ._llcrypt unregistered [11168.277632] Key type ._llcrypt registered [11168.280623] Key type .llcrypt registered [11168.395810] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11170.332133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11170.401880] alg: No test for adler32 (adler32-zlib) [11171.594443] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [11171.947874] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [11173.743702] Key type lgssc registered [11175.265631] Lustre: Echo OBD driver; http://www.lustre.org/ [11185.432916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11185.464113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11187.048042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11193.833199] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11196.407113] LustreError: 358653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11196.432471] LustreError: 358653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11201.505711] LustreError: 358654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11203.775590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11204.303669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11210.035245] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11218.368183] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11225.110401] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11233.653927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11234.508922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11243.123219] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11252.240753] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11257.043799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11270.627518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11270.639082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11270.656537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11272.674256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11272.676674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11272.702337] Lustre: Skipped 1 previous similar message [11272.715541] Lustre: Skipped 2 previous similar messages [11275.637204] Lustre: server umount lustre-MDT0000 complete [11277.802991] LustreError: 359383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11277.815855] LustreError: 359383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11280.459507] LustreError: 360498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129909 with bad export cookie 1443843284148655423 [11280.469857] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11280.481411] LustreError: 360498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11281.371597] Lustre: server umount lustre-MDT0001 complete [11297.224303] Lustre: server umount lustre-OST0000 complete [11313.143574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11313.744577] LustreError: 362390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11313.854037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11319.279719] LustreError: 362391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11319.571736] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11328.486538] LustreError: 362390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11328.513720] LustreError: 362390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11330.108790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11330.560266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11335.983814] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11345.080910] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11351.081538] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11358.394189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11358.751120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11365.897584] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11374.521800] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11383.285482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11398.624749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11398.629949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11398.634454] Lustre: Skipped 1 previous similar message [11398.651930] Lustre: Skipped 2 previous similar messages [11403.747053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11403.758142] Lustre: Skipped 2 previous similar messages [11404.406730] Lustre: server umount lustre-MDT0000 complete [11408.876436] LustreError: 362387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11408.908910] LustreError: 362387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11409.499048] LustreError: 363126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130038 with bad export cookie 1443843284148656529 [11409.504169] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11409.511297] LustreError: 363126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11409.955527] Lustre: server umount lustre-MDT0001 complete [11425.476259] Lustre: server umount lustre-OST0000 complete [11443.504588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11444.316598] LustreError: 366337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11444.444059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11450.568602] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11461.577755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11467.760720] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11477.323991] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11484.198336] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11491.629878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11492.239704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11492.247250] Lustre: Skipped 1 previous similar message [11499.913489] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11509.456713] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11514.842913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11522.016317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11522.019507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11522.072679] Lustre: Skipped 1 previous similar message [11522.086983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11523.496838] Lustre: server umount lustre-OST0000 complete [11530.209062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11530.217405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11530.247819] Lustre: Skipped 1 previous similar message [11538.557891] Lustre: server umount lustre-MDT0000 complete [11539.425587] LustreError: 366337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11539.447807] LustreError: 366337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11542.510049] LustreError: 366319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130171 with bad export cookie 1443843284148657558 [11542.519983] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11542.528524] LustreError: 366319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11543.267945] Lustre: server umount lustre-MDT0001 complete [11558.455866] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [11562.212944] Key type lgssc unregistered [11562.570910] LNet: 369878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11562.581168] LNetError: 369878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11562.598212] LNet: Removed LNI 192.168.202.151@tcp [11563.928350] Key type .llcrypt unregistered [11563.930590] Key type ._llcrypt unregistered [11594.562530] Key type ._llcrypt registered [11594.566347] Key type .llcrypt registered [11594.725543] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11596.531342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11596.559079] alg: No test for adler32 (adler32-zlib) [11597.813400] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [11598.136684] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [11599.903417] Key type lgssc registered [11601.403776] Lustre: Echo OBD driver; http://www.lustre.org/ [11610.152096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11610.191074] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11610.201254] LustreError: 371321:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11610.222983] LustreError: 371321:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11610.239644] LustreError: 371321:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11610.249220] LustreError: 371321:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11610.261098] LustreError: 371321:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11618.494552] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11618.500499] LustreError: 371719:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11618.521146] LustreError: 371719:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11618.526759] LustreError: 371719:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11618.532978] LustreError: 371719:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11618.541996] LustreError: 371719:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11642.247756] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [11646.545991] Key type lgssc unregistered [11647.031827] LNet: 372726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11647.041847] LNetError: 372726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11647.061805] LNet: Removed LNI 192.168.202.151@tcp [11648.058779] Key type .llcrypt unregistered [11648.060945] Key type ._llcrypt unregistered [11672.308546] Key type ._llcrypt registered [11672.310942] Key type .llcrypt registered [11672.416731] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [11691.478805] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11693.255586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11693.450135] alg: No test for adler32 (adler32-zlib) [11694.902132] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [11695.279516] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [11697.143292] Key type lgssc registered [11698.773545] Lustre: Echo OBD driver; http://www.lustre.org/ [11741.116149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11741.140994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11742.398436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11742.475449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11742.611659] Lustre: lustre-MDT0000: new disk, initializing [11742.710896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11742.731280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11748.225907] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11760.895607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11761.112593] Lustre: 375887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11761.235250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11761.242710] Lustre: Skipped 1 previous similar message [11761.401365] Lustre: lustre-MDT0001: new disk, initializing [11761.566260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11761.605200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11761.625418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11766.835774] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11777.543566] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11784.530347] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11792.058320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11792.380744] Lustre: lustre-OST0000: new disk, initializing [11792.389245] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11792.395762] Lustre: 377856:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11792.473296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11799.130438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11799.153188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11799.280536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11800.540675] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11812.328640] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11817.958903] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11818.171972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11822.928673] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11823.300379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11830.240736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11830.248269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11830.258655] LustreError: Skipped 1 previous similar message [11830.269400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11830.279407] Lustre: Skipped 1 previous similar message [11830.303677] Lustre: Skipped 1 previous similar message [11831.600664] Lustre: server umount lustre-OST0000 complete [11838.435364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11838.455107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11838.483435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11840.480516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11840.481140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11840.517358] Lustre: Skipped 1 previous similar message [11842.164808] Lustre: server umount lustre-MDT0000 complete [11845.605169] LustreError: 375894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11845.630636] LustreError: 375894:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11846.043392] LustreError: 376682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130474 with bad export cookie 13406663866710940801 [11846.047582] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11846.055350] LustreError: 376682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11846.474614] Lustre: server umount lustre-MDT0001 complete [11855.970584] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11857.902197] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:08:04 (1787130484) [11865.801113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11866.429114] LustreError: 380318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.525660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11871.717361] LustreError: 380319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11872.035269] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11876.835639] LustreError: 380318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11881.849291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11881.954189] LustreError: 380319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11882.519377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11887.488261] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11896.071678] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11902.382183] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11909.291782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11909.736167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11916.731695] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11925.128278] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11933.214923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11933.532705] Lustre: lustre-OST0001: new disk, initializing [11933.536813] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11933.541937] Lustre: 383153:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11933.658806] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11941.908367] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11944.064586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11944.078364] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11944.221912] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11953.338961] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11959.779587] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11959.792162] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11959.804837] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11959.822025] Lustre: Skipped 1 previous similar message [11959.830492] Lustre: Skipped 1 previous similar message [11964.092710] Lustre: server umount lustre-OST0001 complete [11964.900990] LustreError: 382142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11964.922328] LustreError: 382142:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11983.841147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11983.844282] LustreError: Skipped 1 previous similar message [11983.846490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11983.856230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11983.856238] Lustre: Skipped 1 previous similar message [11983.888228] Lustre: Skipped 1 previous similar message [11985.393905] LustreError: 382141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11985.423131] LustreError: 382141:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [11995.625589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11995.638809] Lustre: Skipped 6 previous similar messages [11997.151459] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11997.348383] Lustre: server umount lustre-OST0000 complete [12005.347506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12005.365380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12008.478585] Lustre: server umount lustre-MDT0000 complete [12012.699788] LustreError: 380299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130641 with bad export cookie 13406663866710941683 [12012.701577] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12012.705942] LustreError: 380299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12019.216808] Lustre: server umount lustre-MDT0001 complete [12033.763529] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12037.593399] Key type lgssc unregistered [12038.020758] LNet: 385153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12038.025984] LNetError: 385153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12038.039198] LNet: Removed LNI 192.168.202.151@tcp [12038.950195] Key type .llcrypt unregistered [12038.952239] Key type ._llcrypt unregistered [12096.304714] Key type ._llcrypt registered [12096.306143] Key type .llcrypt registered [12096.445280] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12098.079698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12098.191842] alg: No test for adler32 (adler32-zlib) [12099.772538] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [12100.139438] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12101.848682] Key type lgssc registered [12103.381627] Lustre: Echo OBD driver; http://www.lustre.org/ [12113.559344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12113.587929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12115.066937] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12115.086442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12115.256541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12121.073705] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12130.850493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12131.053691] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12131.123265] Lustre: 387953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12131.480283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12136.465401] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12145.220344] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12150.656670] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12156.385560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12156.412502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12156.431987] Lustre: Skipped 1 previous similar message [12159.491666] Lustre: server umount lustre-MDT0000 complete [12163.982352] LustreError: 387944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130792 with bad export cookie 15386542353668418655 [12163.992211] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12163.998371] LustreError: 387944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12164.479904] Lustre: server umount lustre-MDT0001 complete [12175.742896] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:13:21 (1787130801) [12183.541055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12184.146428] LustreError: 390465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12184.158967] LustreError: 390465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12184.196547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12189.664807] LustreError: 390466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12189.800714] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12194.794197] LustreError: 390465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12199.341913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12199.982083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12205.501925] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12213.858649] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12220.816865] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12228.646217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12228.906469] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12229.107044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12235.674753] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12244.879546] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12253.412521] Lustre: Permanently deactivating lustre-OST0000 [12253.430779] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12263.076585] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12264.123729] LustreError: 393449:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12264.128620] LustreError: 393449:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12264.130888] LustreError: 393449:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12267.690891] LustreError: 393498:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12267.702524] LustreError: 393498:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12276.286519] Lustre: server umount lustre-OST0000 complete [12280.272647] LustreError: 393748:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12280.278656] LustreError: 393748:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12286.439937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12286.441606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12286.465892] Lustre: Skipped 2 previous similar messages [12286.481944] Lustre: Skipped 1 previous similar message [12291.553608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12291.557349] Lustre: Skipped 1 previous similar message [12292.684262] Lustre: server umount lustre-MDT0000 complete [12296.679699] LustreError: 391191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12296.697536] LustreError: 391191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12296.780573] LustreError: 390446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130925 with bad export cookie 15386542353668419236 [12296.789799] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12296.790355] LustreError: 390446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12306.727620] Lustre: server umount lustre-MDT0001 complete [12316.850337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12322.831189] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12326.374347] LustreError: 394908:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12334.193343] Lustre: 395247:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.151@tcp [12336.850847] Lustre: server umount MGS complete [12348.909175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12349.553315] LustreError: 395972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12349.588304] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12349.597313] Lustre: Skipped 1 previous similar message [12349.705407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12355.027188] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12360.160603] LustreError: 395972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12360.182447] LustreError: 395972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12365.607612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12365.850435] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12365.950460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12370.344965] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12379.219193] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12386.333861] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12388.675423] Lustre: Permanently reactivating lustre-OST0000 [12388.686667] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12388.688930] Lustre: Skipped 2 previous similar messages [12418.015416] LustreError: 397799:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12426.377500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12426.755598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12434.168475] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12443.465339] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12451.654338] Lustre: server umount lustre-OST0000 complete [12457.955187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12457.960571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12457.972140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12464.610517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12464.628127] Lustre: Skipped 3 previous similar messages [12465.715523] Lustre: server umount lustre-MDT0000 complete [12469.727123] Lustre: 387275:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787131082/real 1787131082] req@00000000985f3085 x1873942391395328/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787131098 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12469.733797] LustreError: 395973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12469.779520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12469.779538] Lustre: Skipped 1 previous similar message [12469.853419] LustreError: 395973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12470.171154] LustreError: 395953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131099 with bad export cookie 15386542353668420286 [12470.186899] LustreError: 395953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12470.193960] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12470.590774] Lustre: server umount lustre-MDT0001 complete [12485.101453] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12488.838295] Key type lgssc unregistered [12489.276209] LNet: 400343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12489.288443] LNetError: 400343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12490.342091] LNet: Removed LNI 192.168.202.151@tcp [12491.507247] Key type .llcrypt unregistered [12491.509030] Key type ._llcrypt unregistered [12514.564050] Key type ._llcrypt registered [12514.567286] Key type .llcrypt registered [12514.722632] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [12534.168923] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12535.523350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12535.594245] alg: No test for adler32 (adler32-zlib) [12536.720143] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [12536.986251] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12538.839290] Key type lgssc registered [12540.385653] Lustre: Echo OBD driver; http://www.lustre.org/ [12579.854374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12579.881143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12581.274311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12581.315335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12581.396297] Lustre: lustre-MDT0000: new disk, initializing [12581.556177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12581.592427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12587.329428] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12599.469366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12599.692177] Lustre: 403505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12599.757596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12599.764251] Lustre: Skipped 1 previous similar message [12599.845243] Lustre: lustre-MDT0001: new disk, initializing [12599.999352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12600.043486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12600.062075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12605.488649] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12616.721898] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12623.735807] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12632.125732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12632.498953] Lustre: lustre-OST0000: new disk, initializing [12632.507236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12632.517731] Lustre: 405474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12632.616506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12639.898665] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12641.401679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12641.410958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12641.558127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12651.208754] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12656.868688] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12657.070499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12662.468557] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12662.766960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12667.374670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12667.376158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12667.409535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12671.434879] Lustre: server umount lustre-OST0000 complete [12677.088275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12677.105822] LustreError: Skipped 1 previous similar message [12677.108074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12677.115637] Lustre: Skipped 1 previous similar message [12677.119699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12677.128320] Lustre: Skipped 1 previous similar message [12681.830872] Lustre: server umount lustre-MDT0000 complete [12686.707214] LustreError: 403496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131315 with bad export cookie 5573664388892103354 [12686.710355] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12686.724627] LustreError: 403496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12687.229051] Lustre: server umount lustre-MDT0001 complete [12697.484897] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12699.254100] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:22:05 (1787131325) [12706.532235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12707.097417] LustreError: 407934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12707.135818] LustreError: 407934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12707.191060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12712.429626] LustreError: 407935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12713.211545] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12717.539152] LustreError: 407934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12722.658505] LustreError: 407935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12723.022114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12723.759746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12728.151516] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12736.293779] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12742.958905] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12750.884817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12751.311600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12758.496618] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12768.013294] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12770.337966] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12774.105087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12783.074591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12783.081355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12783.105753] Lustre: Skipped 1 previous similar message [12783.113536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12783.127376] Lustre: Skipped 1 previous similar message [12789.225106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12789.231203] Lustre: Skipped 3 previous similar messages [12794.335288] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12794.338378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12794.352890] Lustre: Skipped 1 previous similar message [12794.533953] Lustre: server umount lustre-OST0000 complete [12799.457610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12799.471027] Lustre: Skipped 2 previous similar messages [12804.579313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12804.584607] Lustre: Skipped 3 previous similar messages [12804.951637] Lustre: server umount lustre-MDT0000 complete [12808.853943] LustreError: 407914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131437 with bad export cookie 5573664388892104236 [12808.865071] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12809.167908] Lustre: server umount lustre-MDT0001 complete [12819.456511] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12822.942817] Key type lgssc unregistered [12823.311923] LNet: 411666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12823.316650] LNetError: 411666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12823.336806] LNet: Removed LNI 192.168.202.151@tcp [12824.490986] Key type .llcrypt unregistered [12824.496602] Key type ._llcrypt unregistered [12849.498947] Key type ._llcrypt registered [12849.501881] Key type .llcrypt registered [12849.618661] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12849.925904] Key type .llcrypt unregistered [12849.931994] Key type ._llcrypt unregistered [12872.047445] Key type ._llcrypt registered [12872.055254] Key type .llcrypt registered [12872.199422] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12874.434785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12874.516627] alg: No test for adler32 (adler32-zlib) [12875.650960] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [12875.889364] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12877.639186] Key type lgssc registered [12879.120712] Lustre: Echo OBD driver; http://www.lustre.org/ [12888.237389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12888.288199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12889.879845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12895.168840] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12899.362088] LustreError: 413888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12899.405771] LustreError: 413888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12904.417907] LustreError: 413889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12904.994480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12905.467732] LustreError: 413888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12905.507805] LustreError: 413888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12905.517423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12910.566988] LustreError: 413889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12910.593734] LustreError: 413889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12913.096801] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12914.660026] LustreError: 413888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12914.682919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12914.693555] LustreError: 413888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12922.707802] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12924.897802] LustreError: 413888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12924.914596] LustreError: 413888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12930.695437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12931.271733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12935.905324] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12945.377749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12945.388672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12945.412033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12946.402935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12946.413332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12946.425218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12946.965694] Lustre: server umount lustre-OST0000 complete [12951.255072] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12951.263617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12951.278131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12951.581442] Lustre: server umount lustre-MDT0000 complete [12952.032963] LustreError: 413889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12952.046220] LustreError: 413889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12955.806649] LustreError: 415995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131584 with bad export cookie 12046300922436835136 [12955.812597] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12955.840614] LustreError: 415995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12958.324485] Lustre: server umount lustre-MDT0001 complete [12969.325426] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [12972.564902] Key type lgssc unregistered [12972.901958] LNet: 417037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12972.908060] LNetError: 417037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12972.930718] LNet: Removed LNI 192.168.202.151@tcp [12973.883572] Key type .llcrypt unregistered [12973.887417] Key type ._llcrypt unregistered [13001.307433] Key type ._llcrypt registered [13001.310480] Key type .llcrypt registered [13001.441858] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [13003.554164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13003.635388] alg: No test for adler32 (adler32-zlib) [13004.853870] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [13005.052291] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [13006.767575] Key type lgssc registered [13007.874344] Lustre: Echo OBD driver; http://www.lustre.org/ [13016.835919] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13016.863778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13032.543560] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13037.663680] LustreError: 418428:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.151@tcp: failed processing log, type 4: rc = -110 [13064.352658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13071.643046] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13080.293066] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13088.547673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13089.298852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13089.429859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13094.369439] LustreError: 418454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13094.376033] LustreError: 418454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13094.445965] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13098.474978] LustreError: 418455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13098.492705] LustreError: 418455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13103.597384] LustreError: 419467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13103.630672] LustreError: 419467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13105.036791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13105.437345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13112.327362] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13123.986486] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13131.365615] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13139.444467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13139.499567] LustreError: 418444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13139.674068] Lustre: server umount lustre-OST0000 complete [13144.394969] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13144.408733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13144.426329] Lustre: Skipped 1 previous similar message [13144.439786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13144.882673] Lustre: server umount lustre-MDT0000 complete [13149.271682] LustreError: 418436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131778 with bad export cookie 11863857424911387865 [13149.281588] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13149.763621] Lustre: server umount lustre-MDT0001 complete [13164.803192] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13169.879534] Key type lgssc unregistered [13170.271362] LNet: 422001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13170.276096] LNetError: 422001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13171.322367] LNet: Removed LNI 192.168.202.151@tcp [13172.664899] Key type .llcrypt unregistered [13172.668963] Key type ._llcrypt unregistered [13202.897519] Key type ._llcrypt registered [13202.905096] Key type .llcrypt registered [13203.214906] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [13205.735181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13205.837536] alg: No test for adler32 (adler32-zlib) [13207.125433] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [13207.643949] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [13209.431163] Key type lgssc registered [13210.629039] Lustre: Echo OBD driver; http://www.lustre.org/ [13218.103808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13218.130533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13219.470960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13223.207550] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13230.055098] LustreError: 423419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13230.081489] LustreError: 423419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13230.118126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13230.512437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13235.473785] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13243.472935] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13249.899603] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13256.594867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13257.064792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13264.170110] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13264.263077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13272.567964] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13276.645369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13276.650888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13276.661393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13279.470889] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.51@tcp (stopping) [13280.739798] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13280.742133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13280.752876] Lustre: Skipped 1 previous similar message [13280.764934] Lustre: Skipped 2 previous similar messages [13282.593537] Lustre: server umount lustre-MDT0000 complete [13284.575505] LustreError: 423414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13285.365674] LustreError: 423401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.51@tcp arrived at 1787131914 with bad export cookie 2941336333221349668 [13285.380253] LustreError: 423401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13285.863177] LustreError: 423420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13285.890264] LustreError: 423420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13287.709240] LustreError: 423400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131916 with bad export cookie 2941336333221348779 [13287.729109] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13290.980276] LustreError: 423416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13290.991194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13291.001060] LustreError: 423416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13291.029449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13291.035944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13294.017590] Lustre: server umount lustre-OST0000 complete [13297.186369] LustreError: 423419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13297.201128] LustreError: 423419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13301.435028] Lustre: server umount lustre-MDT0001 complete [13314.129024] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13317.010777] Key type lgssc unregistered [13317.356683] LNet: 427105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13317.384150] LNetError: 427105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13317.401914] LNet: Removed LNI 192.168.202.151@tcp [13318.818953] Key type .llcrypt unregistered [13318.821775] Key type ._llcrypt unregistered [13344.674256] Key type ._llcrypt registered [13344.678478] Key type .llcrypt registered [13344.876183] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [13346.059686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13346.187995] alg: No test for adler32 (adler32-zlib) [13347.264259] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [13347.502796] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [13349.295214] Key type lgssc registered [13351.283916] Lustre: Echo OBD driver; http://www.lustre.org/ [13360.036452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13360.058929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13361.901530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13368.672315] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13371.364627] LustreError: 428523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13371.389866] LustreError: 428523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13376.500226] LustreError: 428522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13378.507366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13378.960771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13384.310244] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13393.387865] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13399.066536] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13408.041534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13408.782989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13417.154748] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13426.055803] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13431.313810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13436.393461] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13436.396466] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13436.409715] Lustre: Skipped 1 previous similar message [13436.431448] Lustre: Skipped 1 previous similar message [13438.188770] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [13441.506598] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13441.521209] Lustre: Skipped 1 previous similar message [13442.182392] Lustre: server umount lustre-MDT0001 complete [13443.278347] LustreError: 429492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13446.640331] LustreError: 431017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13446.683448] LustreError: 431017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13451.745138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13451.750187] LustreError: 428522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13451.787754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13451.816974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13456.867317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13457.635641] Lustre: server umount lustre-OST0000 complete [13461.984357] LustreError: 428522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13462.032760] LustreError: 428522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13463.857200] Lustre: server umount lustre-MDT0000 complete [13478.159706] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13480.379948] Key type lgssc unregistered [13480.653768] LNet: 432217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13480.658836] LNetError: 432217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13480.675225] LNet: Removed LNI 192.168.202.151@tcp [13481.881519] Key type .llcrypt unregistered [13481.885903] Key type ._llcrypt unregistered [13512.155705] Key type ._llcrypt registered [13512.157968] Key type .llcrypt registered [13512.287185] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [13529.493869] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [13531.058112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13531.082352] alg: No test for adler32 (adler32-zlib) [13532.195866] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [13532.469572] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [13534.207967] Key type lgssc registered [13535.708707] Lustre: Echo OBD driver; http://www.lustre.org/ [13580.396495] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [13599.295304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13599.309337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13600.482843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13600.504665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13600.561266] Lustre: lustre-MDT0000: new disk, initializing [13600.645967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13600.681899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13605.632530] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13621.017700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13621.246484] Lustre: lustre-OST0000: new disk, initializing [13621.249554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13621.255817] Lustre: Skipped 1 previous similar message [13621.260346] Lustre: 437280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13621.327118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13629.576649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13629.593399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13629.651972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13630.791130] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13641.697525] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13659.039795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13659.258438] Lustre: 436260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13659.434989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13659.571165] Lustre: lustre-MDT0001: new disk, initializing [13659.727513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13659.775738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13659.793678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13666.922556] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13677.349617] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13677.626920] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13683.079749] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13683.328642] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13689.317099] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13689.319121] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13689.339853] Lustre: Skipped 1 previous similar message [13689.358251] Lustre: Skipped 1 previous similar message [13690.099735] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [13694.441201] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13694.458232] Lustre: Skipped 1 previous similar message [13694.929115] Lustre: server umount lustre-MDT0001 complete [13695.192658] LustreError: 437943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13695.209876] LustreError: 437943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13699.557323] LustreError: 436273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13699.578827] LustreError: 436273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13704.673424] LustreError: 436269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13704.692498] LustreError: 436269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13709.802453] LustreError: 436273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13709.818812] LustreError: 436273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13714.913564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13714.915396] LustreError: 436274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13714.943532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13720.014957] Lustre: server umount lustre-OST0000 complete [13730.399318] Lustre: server umount lustre-MDT0000 complete [13744.239906] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13746.593986] Key type lgssc unregistered [13746.815753] LNet: 441030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13746.821859] LNetError: 441030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13746.838906] LNet: Removed LNI 192.168.202.151@tcp [13747.656393] Key type .llcrypt unregistered [13747.658855] Key type ._llcrypt unregistered [13786.870888] Key type ._llcrypt registered [13786.874225] Key type .llcrypt registered [13786.992474] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13787.290833] Key type .llcrypt unregistered [13787.292829] Key type ._llcrypt unregistered [13804.316409] Key type ._llcrypt registered [13804.318454] Key type .llcrypt registered [13804.434702] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [13805.719663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13805.797625] alg: No test for adler32 (adler32-zlib) [13807.268486] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [13807.676520] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [13809.447176] Key type lgssc registered [13811.044391] Lustre: Echo OBD driver; http://www.lustre.org/ [13845.581804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13845.628678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13847.021746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13847.074962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13847.134161] Lustre: lustre-MDT0000: new disk, initializing [13847.262512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13847.295795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13852.307896] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13863.864868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13864.098613] Lustre: 444414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13864.180541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13864.183911] Lustre: Skipped 1 previous similar message [13864.277956] Lustre: lustre-MDT0001: new disk, initializing [13864.396414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13864.463645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13864.478731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13869.723561] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13881.214965] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13887.281705] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13895.312292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13895.684523] Lustre: lustre-OST0000: new disk, initializing [13895.686942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13895.693231] Lustre: 446385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13895.826826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13904.437200] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13905.018825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13905.045332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13905.156060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13914.088532] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13925.878570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13925.885215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13925.893160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13925.934209] Lustre: Skipped 1 previous similar message [13928.470356] Lustre: server umount lustre-MDT0000 complete [13932.468135] LustreError: 446384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787132561 with bad export cookie 13295455772395108900 [13932.473127] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13932.486096] LustreError: 446384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13932.783364] Lustre: server umount lustre-MDT0001 complete [13946.431255] Lustre: server umount lustre-OST0000 complete [13958.605611] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:43:04 (1787132584) [13967.066593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13967.647050] LustreError: 448412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13967.664433] LustreError: 448412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13967.760571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13972.963514] LustreError: 448413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13973.668405] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13978.081697] LustreError: 448412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13983.127870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13983.202184] LustreError: 448413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13983.499485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13987.732656] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [13995.598325] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14001.788675] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14022.318472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14022.622315] Lustre: Found index 0 for lustre-OST0000, updating log [14022.819859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14029.554485] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14037.013522] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14038.204277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14048.742633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14048.745956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14048.751290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14048.751298] Lustre: Skipped 2 previous similar messages [14048.768985] Lustre: Skipped 2 previous similar messages [14049.832913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14049.836571] Lustre: Skipped 1 previous similar message [14052.632605] Lustre: server umount lustre-OST0000 complete [14070.755387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14070.795838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14070.817879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14070.820883] Lustre: Skipped 1 previous similar message [14075.048864] Lustre: server umount lustre-MDT0000 complete [14075.363664] LustreError: 448412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14075.383289] LustreError: 448412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14078.786061] LustreError: 448392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787132707 with bad export cookie 13295455772395110293 [14078.787188] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14078.797691] LustreError: 448392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14079.201210] Lustre: server umount lustre-MDT0001 complete [14083.231180] LNet: 1 local NIs in recovery (showing 1): 192.168.202.151@tcp [14088.251829] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:45:14 (1787132714) [14712.829505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14713.505948] LustreError: 454826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14713.521882] LustreError: 454826:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14713.603989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14718.263572] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14718.948413] LustreError: 454827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14724.089809] LustreError: 454826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14729.229834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14729.825093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14736.268807] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14746.626183] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14753.465935] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14774.573605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14774.626582] LNet: 1 local NIs in recovery (showing 1): 192.168.202.151@tcp [14774.947784] Lustre: Found index 0 for lustre-OST0000, updating log [14775.229284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14783.038243] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [14791.987057] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14795.112251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14795.123072] Lustre: Skipped 1 previous similar message [14795.125516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14795.130148] Lustre: Skipped 2 previous similar messages [14800.516341] Lustre: server umount lustre-OST0000 complete [14805.730194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14805.745685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14806.763138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14806.771274] Lustre: Skipped 2 previous similar messages [14808.803830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14808.817504] Lustre: Skipped 3 previous similar messages [14812.905029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14812.912975] Lustre: Skipped 7 previous similar messages [14819.295299] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14819.649807] Lustre: server umount lustre-MDT0000 complete [14820.069708] LustreError: 455742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14820.097248] LustreError: 455742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14823.680844] LustreError: 454807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787133452 with bad export cookie 13295455772395111378 [14823.689770] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14824.006136] Lustre: server umount lustre-MDT0001 complete [14835.103310] LNet: 1 local NIs in recovery (showing 1): 192.168.202.151@tcp [14855.427379] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [14867.131976] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [14907.007653] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:58:53 (1787133533) [14927.128554] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [14930.424857] Key type lgssc unregistered [14930.734330] LNet: 461443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14930.740533] LNetError: 461443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15038.309802] LNet: Removed LNI 192.168.202.151@tcp [15039.385976] Key type .llcrypt unregistered [15039.387578] Key type ._llcrypt unregistered [15059.347622] Key type ._llcrypt registered [15059.353964] Key type .llcrypt registered [15059.593332] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [15061.412592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15061.559546] alg: No test for adler32 (adler32-zlib) [15063.233262] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [15063.664256] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [15065.327128] Key type lgssc registered [15066.842785] Lustre: Echo OBD driver; http://www.lustre.org/ [15072.705026] LNet: Added LNI 192.168.202.151@tcp100 [8/256/0/180] [15073.767498] LNet: Added LNI 192.168.202.151@tcp101 [8/256/0/180] [15074.781420] LNet: Added LNI 192.168.202.151@tcp102 [8/256/0/180] [15075.851388] LNet: Added LNI 192.168.202.151@tcp103 [8/256/0/180] [15076.700945] LNet: Added LNI 192.168.202.151@tcp104 [8/256/0/180] [15077.692820] LNet: Added LNI 192.168.202.151@tcp105 [8/256/0/180] [15078.668659] LNet: Added LNI 192.168.202.151@tcp106 [8/256/0/180] [15079.841582] LNet: Added LNI 192.168.202.151@tcp107 [8/256/0/180] [15080.875544] LNet: Added LNI 192.168.202.151@tcp108 [8/256/0/180] [15082.038887] LNet: Added LNI 192.168.202.151@tcp109 [8/256/0/180] [15083.018041] LNet: Added LNI 192.168.202.151@tcp110 [8/256/0/180] [15083.976085] LNet: Added LNI 192.168.202.151@tcp111 [8/256/0/180] [15085.041513] LNet: Added LNI 192.168.202.151@tcp112 [8/256/0/180] [15086.146790] LNet: Added LNI 192.168.202.151@tcp113 [8/256/0/180] [15087.285642] LNet: Added LNI 192.168.202.151@tcp114 [8/256/0/180] [15088.311920] LNet: Added LNI 192.168.202.151@tcp115 [8/256/0/180] [15089.317036] LNet: Added LNI 192.168.202.151@tcp116 [8/256/0/180] [15090.264625] LNet: Added LNI 192.168.202.151@tcp117 [8/256/0/180] [15091.419338] LNet: Added LNI 192.168.202.151@tcp118 [8/256/0/180] [15092.188868] LNet: Added LNI 192.168.202.151@tcp119 [8/256/0/180] [15093.141169] LNet: Added LNI 192.168.202.151@tcp120 [8/256/0/180] [15094.051767] LNet: Added LNI 192.168.202.151@tcp121 [8/256/0/180] [15095.042058] LNet: Added LNI 192.168.202.151@tcp122 [8/256/0/180] [15096.011595] LNet: Added LNI 192.168.202.151@tcp123 [8/256/0/180] [15097.063192] LNet: Added LNI 192.168.202.151@tcp124 [8/256/0/180] [15098.009621] LNet: Added LNI 192.168.202.151@tcp125 [8/256/0/180] [15098.899606] LNet: Added LNI 192.168.202.151@tcp126 [8/256/0/180] [15100.206913] LNet: Added LNI 192.168.202.151@tcp127 [8/256/0/180] [15101.226855] LNet: Added LNI 192.168.202.151@tcp128 [8/256/0/180] [15102.370738] LNet: Added LNI 192.168.202.151@tcp129 [8/256/0/180] [15103.444310] LNet: Added LNI 192.168.202.151@tcp130 [8/256/0/180] [15104.493769] LNet: Added LNI 192.168.202.151@tcp131 [8/256/0/180] [15105.660617] LNet: Added LNI 192.168.202.151@tcp132 [8/256/0/180] [15106.806344] LNet: Added LNI 192.168.202.151@tcp133 [8/256/0/180] [15107.758795] LNet: Added LNI 192.168.202.151@tcp134 [8/256/0/180] [15108.676798] LNet: Added LNI 192.168.202.151@tcp135 [8/256/0/180] [15117.429970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15117.463243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15118.731236] Lustre: 464499:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [15118.938241] LNetError: 464524:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [15118.957106] LNetError: Protocol error connecting to 192.168.202.151@tcp118 at host 192.168.202.151:988: is it running a compatible version of Lustre? [15118.983273] LNet: 3 local NIs in recovery (showing 3): 192.168.202.151@tcp126, 192.168.202.151@tcp125, 192.168.202.151@tcp128 [15118.984761] LNetError: Skipped 6 previous similar messages [15119.142601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15119.177618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15119.333271] Lustre: lustre-MDT0000: new disk, initializing [15119.451405] Lustre: 464555:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [15119.472557] Lustre: 464555:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15119.515324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15119.547823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15120.032317] LNetError: 461956:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [15120.049571] LNetError: 461956:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 14 previous similar messages [15120.073447] LNetError: Protocol error connecting to 192.168.202.151@tcp126 at host 192.168.202.151:988: is it running a compatible version of Lustre? [15120.095089] LNetError: Skipped 10 previous similar messages [15122.085729] LNetError: 461954:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [15122.098494] LNetError: 461954:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [15122.112582] LNetError: Protocol error connecting to 192.168.202.151@tcp126 at host 192.168.202.151:988: is it running a compatible version of Lustre? [15125.474278] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15126.179616] LNetError: 464530:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [15126.188772] LNetError: Protocol error connecting to 192.168.202.151@tcp126 at host 192.168.202.151:988: is it running a compatible version of Lustre? [15134.370444] LNetError: 461955:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [15134.377834] LNetError: Protocol error connecting to 192.168.202.151@tcp126 at host 192.168.202.151:988: is it running a compatible version of Lustre? [15136.357346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15136.595276] Lustre: 464515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15136.788898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15136.797059] Lustre: Skipped 1 previous similar message [15136.959292] Lustre: lustre-MDT0001: new disk, initializing [15137.053931] Lustre: 465316:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [15137.077242] Lustre: 465316:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15137.274389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15137.348403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15137.356528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15143.721475] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15146.645496] Lustre: 465680:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [15146.656950] Lustre: 465680:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15149.666914] LNetError: 464521:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [15149.674726] LNetError: Protocol error connecting to 192.168.202.151@tcp121 at host 192.168.202.151:988: is it running a compatible version of Lustre? [15154.657817] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15160.007923] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15167.073494] LNetError: 464533:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [15167.090179] LNetError: 464533:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 14 previous similar messages [15167.117972] LNetError: Protocol error connecting to 192.168.202.151@tcp122 at host 192.168.202.151:988: is it running a compatible version of Lustre? [15167.132107] LNetError: Skipped 14 previous similar messages [15167.916978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15168.341424] Lustre: lustre-OST0000: new disk, initializing [15168.344358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15168.352336] Lustre: 466490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15168.531731] Lustre: 466495:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.151@tcp: can't add NID 192.168.202.151@tcp130: rc = -75 [15168.560189] Lustre: 466495:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15168.643066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15176.299083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15176.328822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15176.511264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15176.592592] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15178.464332] LNet: 1 local NIs in recovery (showing 1): 192.168.202.151@tcp126 [15188.496770] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15197.156393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15197.163771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15197.166794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15197.171308] Lustre: Skipped 1 previous similar message [15197.196898] LustreError: Skipped 1 previous similar message [15202.276165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15202.285123] Lustre: Skipped 2 previous similar messages [15203.500304] Lustre: server umount lustre-OST0000 complete [15204.965552] LNetError: 461955:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.151 [15204.974955] LNetError: 461955:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 24 previous similar messages [15204.985169] LNetError: Protocol error connecting to 192.168.202.151@tcp106 at host 192.168.202.151:988: is it running a compatible version of Lustre? [15204.994537] LNetError: Skipped 24 previous similar messages [15208.928069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15208.935997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15208.950324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15212.513304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15212.514444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15214.154381] Lustre: server umount lustre-MDT0000 complete [15217.359144] LustreError: 464508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787133846 with bad export cookie 9586503293408983404 [15217.364446] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15217.369453] LustreError: 464508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15217.741915] Lustre: server umount lustre-MDT0001 complete [15228.287600] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [15231.153486] Key type lgssc unregistered [15231.493698] LNet: 468407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15231.505546] LNetError: 468407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15231.516807] LNet: Removed LNI 192.168.202.151@tcp [15232.543393] LNet: Removed LNI 192.168.202.151@tcp100 [15232.554056] LNetError: 468407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15232.563497] LNetError: 468407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15232.568414] LNet: Removed LNI 192.168.202.151@tcp101 [15233.633599] LNet: Removed LNI 192.168.202.151@tcp102 [15233.636425] LNetError: 468407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15233.644744] LNetError: 468407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15233.654136] LNet: Removed LNI 192.168.202.151@tcp103 [15234.719193] LNet: Removed LNI 192.168.202.151@tcp104 [15235.743188] LNet: Removed LNI 192.168.202.151@tcp105 [15235.749661] LNetError: 468407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15235.767470] LNetError: 468407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15235.785507] LNet: Removed LNI 192.168.202.151@tcp106 [15236.831209] LNet: Removed LNI 192.168.202.151@tcp107 [15237.855384] LNet: Removed LNI 192.168.202.151@tcp108 [15238.879455] LNet: Removed LNI 192.168.202.151@tcp109 [15239.903433] LNet: Removed LNI 192.168.202.151@tcp110 [15239.912837] LNetError: 468407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15239.923708] LNetError: 468407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15239.933574] LNet: Removed LNI 192.168.202.151@tcp111 [15240.993236] LNet: Removed LNI 192.168.202.151@tcp112 [15242.015361] LNet: Removed LNI 192.168.202.151@tcp113 [15243.043821] LNet: Removed LNI 192.168.202.151@tcp114 [15244.063507] LNet: Removed LNI 192.168.202.151@tcp115 [15245.087271] LNet: Removed LNI 192.168.202.151@tcp116 [15246.111170] LNet: Removed LNI 192.168.202.151@tcp117 [15247.135154] LNet: Removed LNI 192.168.202.151@tcp118 [15248.159166] LNet: Removed LNI 192.168.202.151@tcp119 [15248.161443] LNetError: 468407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15248.170114] LNetError: 468407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15248.179233] LNet: Removed LNI 192.168.202.151@tcp120 [15249.247625] LNet: Removed LNI 192.168.202.151@tcp121 [15250.271185] LNet: Removed LNI 192.168.202.151@tcp122 [15251.295299] LNet: Removed LNI 192.168.202.151@tcp123 [15252.321553] LNet: Removed LNI 192.168.202.151@tcp124 [15253.343185] LNet: Removed LNI 192.168.202.151@tcp125 [15253.351943] LNet: Removed LNI 192.168.202.151@tcp126 [15254.367161] LNet: Removed LNI 192.168.202.151@tcp127 [15255.391209] LNet: Removed LNI 192.168.202.151@tcp128 [15256.415250] LNet: Removed LNI 192.168.202.151@tcp129 [15256.418915] LNet: Removed LNI 192.168.202.151@tcp130 [15256.420681] LNet: Removed LNI 192.168.202.151@tcp131 [15256.424106] LNet: Removed LNI 192.168.202.151@tcp132 [15256.430108] LNet: Removed LNI 192.168.202.151@tcp133 [15256.432319] LNet: Removed LNI 192.168.202.151@tcp134 [15256.450972] LNet: Removed LNI 192.168.202.151@tcp135 [15257.031699] Key type .llcrypt unregistered [15257.033547] Key type ._llcrypt unregistered [15282.214774] Key type ._llcrypt registered [15282.216201] Key type .llcrypt registered [15282.297172] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [15299.943540] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [15301.251819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15301.277549] alg: No test for adler32 (adler32-zlib) [15302.486745] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [15303.007655] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [15304.743875] Key type lgssc registered [15306.157514] Lustre: Echo OBD driver; http://www.lustre.org/ [15342.513955] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:06:08 (1787133968) [15362.881334] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [15366.287530] Key type lgssc unregistered [15366.585436] LNet: 472502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15366.589894] LNetError: 472502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15366.601384] LNet: Removed LNI 192.168.202.151@tcp [15367.414669] Key type .llcrypt unregistered [15367.422458] Key type ._llcrypt unregistered [15383.837670] Key type ._llcrypt registered [15383.840779] Key type .llcrypt registered [15383.994597] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [15385.610880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15385.678790] alg: No test for adler32 (adler32-zlib) [15386.907918] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [15387.203178] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [15388.879185] Key type lgssc registered [15389.813409] Lustre: Echo OBD driver; http://www.lustre.org/ [15400.145019] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15400.175101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15401.400775] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15401.419804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15401.494875] Lustre: lustre-MDT0000: new disk, initializing [15401.585464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15401.599040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15406.663890] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15419.251646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15419.511709] Lustre: 473720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15419.651895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15419.662741] Lustre: Skipped 1 previous similar message [15419.770342] Lustre: lustre-MDT0001: new disk, initializing [15419.959231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15420.025360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15420.033561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15426.795477] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15436.949120] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15443.687355] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15452.032415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15452.506654] Lustre: lustre-OST0000: new disk, initializing [15452.515133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15452.528867] Lustre: 475689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15452.639680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15461.594213] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15463.054257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15463.071586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15463.234057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15474.197996] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15479.035314] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing lnet_if_list [15484.503839] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15484.550903] LNet: Added LNI 192.168.202.151@tcp10 [8/256/0/180] [15490.005505] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15490.033299] LNet: Added LNI 192.168.202.151@tcp11 [8/256/0/180] [15494.738982] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15494.803535] LNet: Added LNI 192.168.202.151@tcp12 [8/256/0/180] [15499.494322] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15499.531873] LNet: Added LNI 192.168.202.151@tcp13 [8/256/0/180] [15504.511860] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15504.547854] LNet: Added LNI 192.168.202.151@tcp14 [8/256/0/180] [15509.911381] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15509.945791] LNet: Added LNI 192.168.202.151@tcp15 [8/256/0/180] [15514.876305] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15514.906145] LNet: Added LNI 192.168.202.151@tcp16 [8/256/0/180] [15519.854221] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15519.881339] LNet: Added LNI 192.168.202.151@tcp17 [8/256/0/180] [15525.606368] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15525.626510] LNet: Added LNI 192.168.202.151@tcp18 [8/256/0/180] [15531.023847] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15531.067088] LNet: Added LNI 192.168.202.151@tcp19 [8/256/0/180] [15537.412873] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15537.455842] LNet: Added LNI 192.168.202.151@tcp20 [8/256/0/180] [15544.090290] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15544.137323] LNet: Added LNI 192.168.202.151@tcp21 [8/256/0/180] [15551.636572] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15551.676668] LNet: Added LNI 192.168.202.151@tcp22 [8/256/0/180] [15559.413573] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15559.453476] LNet: Added LNI 192.168.202.151@tcp23 [8/256/0/180] [15565.479891] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15565.545468] LNet: Added LNI 192.168.202.151@tcp24 [8/256/0/180] [15574.357194] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15574.378782] LNet: Added LNI 192.168.202.151@tcp25 [8/256/0/180] [15581.805108] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15581.855286] LNet: Added LNI 192.168.202.151@tcp26 [8/256/0/180] [15588.882591] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15588.911664] LNet: Added LNI 192.168.202.151@tcp27 [8/256/0/180] [15597.139584] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15597.195125] LNet: Added LNI 192.168.202.151@tcp28 [8/256/0/180] [15605.120302] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15605.143791] LNet: Added LNI 192.168.202.151@tcp29 [8/256/0/180] [15611.410810] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15611.475465] LNet: Added LNI 192.168.202.151@tcp30 [8/256/0/180] [15619.953485] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15619.983715] LNet: Added LNI 192.168.202.151@tcp31 [8/256/0/180] [15630.242564] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15630.312390] LNet: Added LNI 192.168.202.151@tcp32 [8/256/0/180] [15639.286310] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15639.314344] LNet: Added LNI 192.168.202.151@tcp33 [8/256/0/180] [15647.402469] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15647.446313] LNet: Added LNI 192.168.202.151@tcp34 [8/256/0/180] [15655.918394] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15655.954581] LNet: Added LNI 192.168.202.151@tcp35 [8/256/0/180] [15664.644203] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15664.682946] LNet: Added LNI 192.168.202.151@tcp36 [8/256/0/180] [15673.836989] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15673.877375] LNet: Added LNI 192.168.202.151@tcp37 [8/256/0/180] [15682.190801] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15682.227565] LNet: Added LNI 192.168.202.151@tcp38 [8/256/0/180] [15689.861336] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15689.898451] LNet: Added LNI 192.168.202.151@tcp39 [8/256/0/180] [15697.469974] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15697.506614] LNet: Added LNI 192.168.202.151@tcp40 [8/256/0/180] [15706.174753] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15706.204068] LNet: Added LNI 192.168.202.151@tcp41 [8/256/0/180] [15716.948885] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15717.009847] LNet: Added LNI 192.168.202.151@tcp42 [8/256/0/180] [15726.876895] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15726.914021] LNet: Added LNI 192.168.202.151@tcp43 [8/256/0/180] [15735.739218] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15735.769863] LNet: Added LNI 192.168.202.151@tcp44 [8/256/0/180] [15744.941401] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15745.013658] LNet: Added LNI 192.168.202.151@tcp45 [8/256/0/180] [15753.838584] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15753.866627] LNet: Added LNI 192.168.202.151@tcp46 [8/256/0/180] [15764.570841] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15764.623695] LNet: Added LNI 192.168.202.151@tcp47 [8/256/0/180] [15774.018746] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15774.044396] LNet: Added LNI 192.168.202.151@tcp48 [8/256/0/180] [15783.391885] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15783.426468] LNet: Added LNI 192.168.202.151@tcp49 [8/256/0/180] [15793.248962] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15793.322473] LNet: Added LNI 192.168.202.151@tcp50 [8/256/0/180] [15814.117331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15814.127534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15814.158533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15816.677954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15816.681429] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15816.721824] Lustre: Skipped 1 previous similar message [15819.702445] Lustre: server umount lustre-MDT0000 complete [15824.742224] LustreError: 473710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787134453 with bad export cookie 5001241161067603121 [15824.748161] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15824.754671] LustreError: 473710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15825.023318] Lustre: server umount lustre-MDT0001 complete [15842.271552] Lustre: 472991:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787134455/real 1787134455] req@00000000e7d92b94 x1873945838151936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787134471 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15842.310417] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15845.343127] Lustre: 472988:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787134458/real 1787134458] req@0000000014686b32 x1873945838152064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787134474 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15847.401295] Lustre: 472989:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787134460/real 1787134460] req@000000008426e5b5 x1873945838152192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787134476 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15849.388423] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [15850.463088] Lustre: 472990:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787134463/real 1787134463] req@00000000b8f2b066 x1873945838152448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787134479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15856.608380] Lustre: 472991:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787134469/real 1787134469] req@00000000021ceca7 x1873945838152576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787134485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15860.366903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15870.434021] LustreError: 475682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15870.463494] LustreError: 475682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15870.574727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15871.600605] LustreError: 475683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15871.646972] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15872.672293] LustreError: 475682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15877.151409] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15878.113171] LustreError: 475683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15883.241622] LustreError: 475682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15889.344300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15889.764287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15895.036190] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15896.150442] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [15905.319995] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15911.092219] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15935.969124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15935.973684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15935.980541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15935.980550] Lustre: Skipped 2 previous similar messages [15936.011275] Lustre: Skipped 1 previous similar message [15939.175212] Lustre: server umount lustre-OST0000 complete [15946.208807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15946.220149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15946.240403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15946.277332] Lustre: Skipped 1 previous similar message [15946.305309] Lustre: Skipped 2 previous similar messages [15949.428074] Lustre: server umount lustre-MDT0000 complete [15951.335424] LustreError: 501219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15951.377122] LustreError: 501219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15954.472823] LustreError: 475688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787134583 with bad export cookie 5001241161067605732 [15954.473689] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15954.485402] LustreError: 475688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15954.963361] Lustre: server umount lustre-MDT0001 complete [15969.579139] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [15973.432911] Key type lgssc unregistered [15973.906328] LNet: 502743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15973.913696] LNetError: 502743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15973.923850] LNet: Removed LNI 192.168.202.151@tcp [15973.928607] LNet: Removed LNI 192.168.202.151@tcp10 [15973.933629] LNet: Removed LNI 192.168.202.151@tcp11 [15973.936642] LNet: Removed LNI 192.168.202.151@tcp12 [15973.941097] LNet: Removed LNI 192.168.202.151@tcp13 [15973.943797] LNet: Removed LNI 192.168.202.151@tcp14 [15973.946844] LNet: Removed LNI 192.168.202.151@tcp15 [15973.952651] LNet: Removed LNI 192.168.202.151@tcp16 [15973.965748] LNet: Removed LNI 192.168.202.151@tcp17 [15973.978834] LNet: Removed LNI 192.168.202.151@tcp18 [15973.984973] LNet: Removed LNI 192.168.202.151@tcp19 [15973.987365] LNet: Removed LNI 192.168.202.151@tcp20 [15973.991462] LNet: Removed LNI 192.168.202.151@tcp21 [15973.994804] LNet: Removed LNI 192.168.202.151@tcp22 [15973.997919] LNet: Removed LNI 192.168.202.151@tcp23 [15974.002500] LNet: Removed LNI 192.168.202.151@tcp24 [15974.005779] LNet: Removed LNI 192.168.202.151@tcp25 [15974.008641] LNet: Removed LNI 192.168.202.151@tcp26 [15974.012420] LNet: Removed LNI 192.168.202.151@tcp27 [15974.015742] LNet: Removed LNI 192.168.202.151@tcp28 [15974.020984] LNet: Removed LNI 192.168.202.151@tcp29 [15974.024763] LNet: Removed LNI 192.168.202.151@tcp30 [15974.027674] LNet: Removed LNI 192.168.202.151@tcp31 [15974.030460] LNet: Removed LNI 192.168.202.151@tcp32 [15974.033200] LNet: Removed LNI 192.168.202.151@tcp33 [15974.043748] LNet: Removed LNI 192.168.202.151@tcp34 [15974.047667] LNet: Removed LNI 192.168.202.151@tcp35 [15974.050762] LNet: Removed LNI 192.168.202.151@tcp36 [15974.056428] LNet: Removed LNI 192.168.202.151@tcp37 [15974.061239] LNet: Removed LNI 192.168.202.151@tcp38 [15974.063712] LNet: Removed LNI 192.168.202.151@tcp39 [15974.067635] LNet: Removed LNI 192.168.202.151@tcp40 [15974.073092] LNet: Removed LNI 192.168.202.151@tcp41 [15974.083943] LNet: Removed LNI 192.168.202.151@tcp42 [15974.089482] LNet: Removed LNI 192.168.202.151@tcp43 [15974.092556] LNet: Removed LNI 192.168.202.151@tcp44 [15974.096421] LNet: Removed LNI 192.168.202.151@tcp45 [15974.099338] LNet: Removed LNI 192.168.202.151@tcp46 [15974.104978] LNet: Removed LNI 192.168.202.151@tcp47 [15974.107567] LNet: Removed LNI 192.168.202.151@tcp48 [15974.118074] LNet: Removed LNI 192.168.202.151@tcp49 [15974.126839] LNet: Removed LNI 192.168.202.151@tcp50 [15975.044788] Key type .llcrypt unregistered [15975.047080] Key type ._llcrypt unregistered [15997.785917] Key type ._llcrypt registered [15997.788910] Key type .llcrypt registered [15997.893449] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [16017.482053] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [16018.547950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16018.569018] alg: No test for adler32 (adler32-zlib) [16020.402993] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [16021.036950] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [16022.818334] Key type lgssc registered [16024.628898] Lustre: Echo OBD driver; http://www.lustre.org/ [16071.340265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16071.373185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16072.728823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16072.774845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16072.871473] Lustre: lustre-MDT0000: new disk, initializing [16073.048855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16073.095533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16078.467873] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16090.674766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16090.884047] Lustre: 505941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16090.977677] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16090.985565] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [16090.998855] Lustre: Skipped 1 previous similar message [16091.149323] Lustre: lustre-MDT0001: new disk, initializing [16091.308996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16091.340299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16091.360643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16096.059312] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16106.356705] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16112.654762] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16120.431968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16120.665306] Lustre: *** cfs_fail_loc=90f, val=0*** [16120.780475] Lustre: lustre-OST0000: new disk, initializing [16120.784232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16120.789775] Lustre: 507957:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16120.879202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16126.042986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16126.054649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16126.155586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16126.532256] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16136.759132] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16146.917836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16146.921506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16146.928039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16146.938717] LustreError: Skipped 1 previous similar message [16146.970338] Lustre: Skipped 1 previous similar message [16152.034138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16152.047138] Lustre: Skipped 2 previous similar messages [16152.971305] Lustre: server umount lustre-OST0000 complete [16157.153248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16157.158324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16157.160431] Lustre: Skipped 1 previous similar message [16157.170586] Lustre: Skipped 1 previous similar message [16162.273884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16162.281721] Lustre: Skipped 1 previous similar message [16163.678981] Lustre: server umount lustre-MDT0000 complete [16168.527300] LustreError: 505934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787134797 with bad export cookie 18380281898094695580 [16168.557358] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16169.389208] Lustre: server umount lustre-MDT0001 complete [16183.697363] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [16186.779634] Key type lgssc unregistered [16187.076900] LNet: 509869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16187.084067] LNetError: 509869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16187.112178] LNet: Removed LNI 192.168.202.151@tcp [16188.433196] Key type .llcrypt unregistered [16188.435776] Key type ._llcrypt unregistered [16212.909698] Key type ._llcrypt registered [16212.911983] Key type .llcrypt registered [16213.021678] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [16214.333815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16214.427290] alg: No test for adler32 (adler32-zlib) [16215.735823] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [16216.169194] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [16218.047206] Key type lgssc registered [16219.396329] Lustre: Echo OBD driver; http://www.lustre.org/ [16228.316995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16228.362558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16229.982274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16234.384678] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16240.120750] LustreError: 511144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16240.136135] LustreError: 511144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16243.804791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16244.436860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16248.938608] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16258.265643] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16263.527129] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16271.019147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16271.341291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16278.571943] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16288.112605] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16301.307805] loop0: detected capacity change from 0 to 204800000 [16301.529446] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16301.549086] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16301.561859] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16310.176698] loop0: detected capacity change from 0 to 204800000 [16310.527397] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16310.539578] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16310.562374] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16318.039875] loop0: detected capacity change from 0 to 204800000 [16325.718214] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16326.014733] Lustre: fs15246-OST0000: new disk, initializing [16326.031250] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16326.043643] Lustre: 514738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16326.099069] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16333.424143] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16344.460316] loop1: detected capacity change from 0 to 204800000 [16350.853166] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16350.964561] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16351.037955] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16351.115177] Lustre: fs15246-MDT0000: new disk, initializing [16351.273215] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16351.303446] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16351.404664] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16352.480762] Lustre: 515860:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000021c8c7da x1873946707283712/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787134997 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16355.176370] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16357.618771] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16357.629418] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16357.668277] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16382.948626] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16382.955236] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16386.242921] Lustre: server umount fs15246-MDT0000 complete [16388.068491] LustreError: 511141:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16393.195742] LustreError: 511141:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16397.204145] Lustre: server umount fs15246-OST0000 complete [16408.552093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16408.555676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16408.556848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16408.597757] Lustre: Skipped 1 previous similar message [16413.678360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16413.699070] Lustre: Skipped 2 previous similar messages [16414.697517] Lustre: server umount lustre-OST0000 complete [16423.394265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16423.410472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16423.446482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16429.028796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16429.031035] Lustre: Skipped 2 previous similar messages [16434.143130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16434.298742] Lustre: server umount lustre-MDT0000 complete [16438.244145] LustreError: 511144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16438.267091] LustreError: 511144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16438.731580] LustreError: 511125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135067 with bad export cookie 9722994727266013304 [16438.749077] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16438.750207] LustreError: 511125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16439.028150] Lustre: server umount lustre-MDT0001 complete [16453.502283] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [16457.350199] Key type lgssc unregistered [16457.653425] LNet: 518963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16457.664944] LNetError: 518963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16458.733992] LNet: Removed LNI 192.168.202.151@tcp [16459.774590] Key type .llcrypt unregistered [16459.779390] Key type ._llcrypt unregistered [16481.646181] Key type ._llcrypt registered [16481.648537] Key type .llcrypt registered [16481.741393] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [16482.782030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16482.908709] alg: No test for adler32 (adler32-zlib) [16484.192788] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [16484.455300] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [16486.231291] Key type lgssc registered [16487.622708] Lustre: Echo OBD driver; http://www.lustre.org/ [16530.847226] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [16541.650537] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [16579.821742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16579.844432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16581.199488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16581.243333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16581.300518] Lustre: lustre-MDT0000: new disk, initializing [16581.375190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16581.392393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16586.755908] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16598.852987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16599.039876] Lustre: 523433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16599.132236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16599.136390] Lustre: Skipped 1 previous similar message [16599.228870] Lustre: lustre-MDT0001: new disk, initializing [16599.368196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16599.449634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16599.470442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16605.022506] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16615.314261] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16622.147545] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16629.773102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16630.041733] Lustre: lustre-OST0000: new disk, initializing [16630.044894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16630.052651] Lustre: 525400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16630.142852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16637.529204] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [16638.071556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16638.085266] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16638.266667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16648.385432] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16653.165845] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16653.391692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16658.195963] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16658.392463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16664.043193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16664.044795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16664.054228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16664.061891] Lustre: Skipped 1 previous similar message [16666.823298] Lustre: server umount lustre-OST0000 complete [16674.272661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16674.283516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16674.286194] Lustre: Skipped 1 previous similar message [16678.010830] Lustre: server umount lustre-MDT0000 complete [16682.262968] LustreError: 523427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787135311 with bad export cookie 5379933996453045176 [16682.277340] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16682.279369] LustreError: 523427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16682.805952] Lustre: server umount lustre-MDT0001 complete [16712.579307] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [16724.115850] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [16769.426356] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16436 sec ========= 06:29:54 (1787135394) [16772.038520] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:29:57 (1787135397) === [16776.665958] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:30:02 (1787135402) === [16829.106718] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [16833.095926] Key type lgssc unregistered [16833.562443] LNet: 532052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16833.571154] LNetError: 532052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16834.599417] LNet: Removed LNI 192.168.202.151@tcp [16835.868796] Key type .llcrypt unregistered [16835.873109] Key type ._llcrypt unregistered