[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 497540369 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003257] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008559] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.012068] LSM: Security Framework initializing [ 0.013054] Yama: becoming mindful. [ 0.015016] SELinux: Initializing. [ 0.016068] *** VALIDATE selinux *** [ 0.024750] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029591] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032117] *** VALIDATE tmpfs *** [ 0.033398] *** VALIDATE proc *** [ 0.034202] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036232] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039025] Spectre V2 : User space: Vulnerable [ 0.040006] Speculative Store Bypass: Vulnerable [ 0.043571] debug: unmapping init [mem 0xffffffff9b059000-0xffffffff9b060fff] [ 0.045184] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046749] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047036] ... version: 2 [ 0.048016] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052019] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054335] rcu: Hierarchical SRCU implementation. [ 0.056521] smp: Bringing up secondary CPUs ... [ 0.057632] x86: Booting SMP configuration: [ 0.058031] .... node #0, CPUs: #1 #2 #3 [ 0.067184] smp: Brought up 1 node, 4 CPUs [ 0.069014] smpboot: Max logical packages: 1 [ 0.070014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.096022] node 0 deferred pages initialised in 24ms [ 0.099111] devtmpfs: initialized [ 0.100284] x86/mm: Memory block size: 128MB [ 0.104073] gcov: version magic: 0x41383552 [ 0.106093] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.110072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.111386] pinctrl core: initialized pinctrl subsystem [ 0.113139] [ 0.113523] ************************************************************* [ 0.115009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.117016] ** ** [ 0.120012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.122012] ** ** [ 0.123009] ** This means that this kernel is built to expose internal ** [ 0.125008] ** IOMMU data structures, which may compromise security on ** [ 0.126008] ** your system. ** [ 0.128009] ** ** [ 0.129008] ** If you see this message and you are not debugging the ** [ 0.130008] ** kernel, report this immediately to your vendor! ** [ 0.132008] ** ** [ 0.133006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135009] ************************************************************* [ 0.136604] NET: Registered protocol family 16 [ 0.138448] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.140057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.142062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.145094] cpuidle: using governor menu [ 0.146668] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.148387] PCI: Using configuration type 1 for base access [ 0.150109] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.157189] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.159079] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.162142] cryptd: max_cpu_qlen set to 1000 [ 0.165215] ACPI: Added _OSI(Module Device) [ 0.166012] ACPI: Added _OSI(Processor Device) [ 0.166984] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.168010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.171702] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.176494] ACPI: Interpreter enabled [ 0.177041] ACPI: PM: (supports S0 S3 S4 S5) [ 0.178011] ACPI: Using IOAPIC for interrupt routing [ 0.179100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.181288] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.189031] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.191038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.193017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.195083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.199312] acpiphp: Slot [2] registered [ 0.200168] acpiphp: Slot [5] registered [ 0.202112] acpiphp: Slot [6] registered [ 0.203114] acpiphp: Slot [7] registered [ 0.205115] acpiphp: Slot [8] registered [ 0.206100] acpiphp: Slot [9] registered [ 0.208123] acpiphp: Slot [10] registered [ 0.209077] acpiphp: Slot [3] registered [ 0.209986] acpiphp: Slot [4] registered [ 0.211090] acpiphp: Slot [11] registered [ 0.212082] acpiphp: Slot [12] registered [ 0.212978] acpiphp: Slot [13] registered [ 0.214060] acpiphp: Slot [14] registered [ 0.217117] acpiphp: Slot [15] registered [ 0.218095] acpiphp: Slot [16] registered [ 0.219117] acpiphp: Slot [17] registered [ 0.221093] acpiphp: Slot [18] registered [ 0.222084] acpiphp: Slot [19] registered [ 0.223081] acpiphp: Slot [20] registered [ 0.224064] acpiphp: Slot [21] registered [ 0.225049] acpiphp: Slot [22] registered [ 0.226047] acpiphp: Slot [23] registered [ 0.227062] acpiphp: Slot [24] registered [ 0.228045] acpiphp: Slot [25] registered [ 0.229041] acpiphp: Slot [26] registered [ 0.231087] acpiphp: Slot [27] registered [ 0.232082] acpiphp: Slot [28] registered [ 0.233080] acpiphp: Slot [29] registered [ 0.234104] acpiphp: Slot [30] registered [ 0.236079] acpiphp: Slot [31] registered [ 0.237052] PCI host bridge to bus 0000:00 [ 0.238017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.240021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.242022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.244018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.246024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.249027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.251174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.252905] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.256339] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.264016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.268872] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.272021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.274022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.276019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.279577] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.283755] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.286052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.289474] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.293012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.305015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.307012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.310937] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.315018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.327018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.341014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.347982] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.357022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.365016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.381018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.390465] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.405020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.410013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.421020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.431392] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.443017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.451019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.467030] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.475268] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.482019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.487017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.499016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.507120] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.511016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.516018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.527021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.536497] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.540504] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.543392] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.545353] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.547217] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.553026] iommu: Default domain type: Passthrough [ 0.554358] SCSI subsystem initialized [ 0.556112] ACPI: bus type USB registered [ 0.557083] usbcore: registered new interface driver usbfs [ 0.558066] usbcore: registered new interface driver hub [ 0.559084] usbcore: registered new device driver usb [ 0.561135] pps_core: LinuxPPS API ver. 1 registered [ 0.562007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.564045] PTP clock support registered [ 0.565151] EDAC MC: Ver: 3.0.0 [ 0.566503] PCI: Using ACPI for IRQ routing [ 0.567833] NetLabel: Initializing [ 0.569014] NetLabel: domain hash size = 128 [ 0.571012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.573086] NetLabel: unlabeled traffic allowed by default [ 0.575070] vgaarb: loaded [ 0.576231] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.577011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.586711] clocksource: Switched to clocksource kvm-clock [ 0.683630] VFS: Disk quotas dquot_6.6.0 [ 0.685734] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.689667] *** VALIDATE ramfs *** [ 0.691302] *** VALIDATE hugetlbfs *** [ 0.693635] pnp: PnP ACPI init [ 0.698223] pnp: PnP ACPI: found 6 devices [ 0.720790] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.723326] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.725176] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.726879] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.728701] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.730687] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.732795] NET: Registered protocol family 2 [ 0.734670] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.738515] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.741326] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.746259] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.750123] TCP: Hash tables configured (established 65536 bind 65536) [ 0.753170] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.756536] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.759326] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.762410] NET: Registered protocol family 1 [ 0.765165] RPC: Registered named UNIX socket transport module. [ 0.767703] RPC: Registered udp transport module. [ 0.769429] RPC: Registered tcp transport module. [ 0.771404] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.774199] NET: Registered protocol family 44 [ 0.776090] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.778785] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.781134] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.783839] PCI: CLS 0 bytes, default 64 [ 0.785507] Unpacking initramfs... [ 2.181451] debug: unmapping init [mem 0xffff9d9a3cc54000-0xffff9d9a3ffbffff] [ 2.185604] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.187675] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.190741] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.703640] Initialise system trusted keyrings [ 2.706059] Key type blacklist registered [ 2.708234] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.729701] zbud: loaded [ 2.737795] *** VALIDATE nfs *** [ 2.738690] *** VALIDATE nfs4 *** [ 2.743613] pstore: using deflate compression [ 2.753521] Platform Keyring initialized [ 3.173191] NET: Registered protocol family 38 [ 3.174846] Key type asymmetric registered [ 3.178109] Asymmetric key parser 'x509' registered [ 3.182773] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.195330] io scheduler mq-deadline registered [ 3.199423] io scheduler kyber registered [ 3.209741] io scheduler bfq registered [ 3.214552] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.224433] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.236439] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.244752] ACPI: Power Button [PWRF] [ 3.250058] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.280296] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.310655] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.323218] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.348278] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.378607] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.408352] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.413335] Non-volatile memory driver v1.3 [ 3.414847] Linux agpgart interface v0.103 [ 3.449194] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.452134] vda: detected capacity change from 0 to 74706944 [ 3.473866] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.476767] vdb: detected capacity change from 0 to 1073741824 [ 3.500664] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.504409] vdc: detected capacity change from 0 to 2621440000 [ 3.522536] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.525773] vdd: detected capacity change from 0 to 2621440000 [ 3.550692] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.553989] vde: detected capacity change from 0 to 4294967296 [ 3.577642] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.583042] vdf: detected capacity change from 0 to 4294967296 [ 3.591284] libphy: Fixed MDIO Bus: probed [ 3.599192] usbcore: registered new interface driver usbserial_generic [ 3.601310] usbserial: USB Serial support registered for generic [ 3.605364] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.611368] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.613757] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.617280] mousedev: PS/2 mouse device common for all mice [ 3.622909] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.626559] rtc_cmos 00:05: RTC can wake from S4 [ 3.627578] rtc_cmos 00:05: registered as rtc0 [ 3.627624] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.627652] intel_pstate: CPU model not supported [ 3.633593] hid: raw HID events driver (C) Jiri Kosina [ 3.644733] usbcore: registered new interface driver usbhid [ 3.648157] usbhid: USB HID core driver [ 3.649545] drop_monitor: Initializing network drop monitor service [ 3.652480] Initializing XFRM netlink socket [ 3.654294] NET: Registered protocol family 10 [ 3.657647] Segment Routing with IPv6 [ 3.660189] NET: Registered protocol family 17 [ 3.662305] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.666886] mpls_gso: MPLS GSO support [ 3.671994] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.678255] RAS: Correctable Errors collector initialized. [ 3.680560] AVX version of gcm_enc/dec engaged. [ 3.682484] AES CTR mode by8 optimization enabled [ 3.810334] sched_clock: Marking stable (3810300266, 0)->(4689188693, -878888427) [ 3.816334] registered taskstats version 1 [ 3.818764] Loading compiled-in X.509 certificates [ 3.821703] zswap: loaded using pool lzo/zbud [ 3.869156] Key type big_key registered [ 3.901778] Key type encrypted registered [ 3.904698] ima: No TPM chip found, activating TPM-bypass! [ 3.917523] ima: Allocated hash algorithm: sha1 [ 3.924740] ima: No architecture policies found [ 3.931341] evm: Initialising EVM extended attributes: [ 3.936713] evm: security.selinux [ 3.939676] evm: security.ima [ 3.941299] evm: security.capability [ 3.943180] evm: HMAC attrs: 0x1 [ 3.946592] rtc_cmos 00:05: setting system clock to 2026-08-14 22:46:38 UTC (1786747598) [ 3.966553] debug: unmapping init [mem 0xffffffff9c003000-0xffffffff9c1fffff] [ 3.974970] debug: unmapping init [mem 0xffffffff9ad82000-0xffffffff9b058fff] [ 3.984177] Write protecting the kernel read-only data: 28672k [ 3.988661] debug: unmapping init [mem 0xffffffff99403000-0xffffffff995fffff] [ 3.993981] debug: unmapping init [mem 0xffffffff99d14000-0xffffffff99dfffff] [ 4.082229] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.105451] systemd[1]: Detected virtualization kvm. [ 4.107021] systemd[1]: Detected architecture x86-64. [ 4.108204] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.218302] systemd[1]: No hostname configured. [ 4.224278] systemd[1]: Set hostname to . [ 4.232773] random: systemd: uninitialized urandom read (16 bytes read) [ 4.245613] systemd[1]: Initializing machine ID from random generator. [ 4.409863] random: ln: uninitialized urandom read (6 bytes read) [ 4.671827] random: systemd: uninitialized urandom read (16 bytes read) [ 4.685398] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.707090] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.758868] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.503371] device-mapper: uevent: version 1.0.3 [ 6.507153] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 7.740844] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.870724] random: fast init done [ 7.922883] scsi host0: ata_piix [ 7.999574] scsi host1: ata_piix [ 8.004400] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.011120] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.458058] random: crng init done [ 13.467975] random: 7 urandom warning(s) missed due to ratelimiting [ 14.624962] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.718374] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ 17.810343] hrtimer: interrupt took 6109953 ns [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.391950] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.584518] SELinux: Disabled at runtime. [ 21.789277] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.814742] systemd[1]: Detected virtualization kvm. [ 21.820659] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.921034] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.933919] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.994394] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.000548] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.014852] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.047753] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.075087] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ 24.204500] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 25.722463] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.191318] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.385964] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.879736] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.966089] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.473490] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 33.199406] NFS: Registering the id_resolver key type [ 33.201608] Key type id_resolver registered [ 33.203638] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg218-server login: [ 94.444153] libcfs: loading out-of-tree module taints kernel. [ 94.489263] Key type ._llcrypt registered [ 94.493189] Key type .llcrypt registered [ 94.634441] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 116.415482] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 118.271225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 118.305096] alg: No test for adler32 (adler32-zlib) [ 120.170557] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 121.302291] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 123.161906] Key type lgssc registered [ 124.721750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.618918] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 196.414654] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 212.008695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 212.063108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.654455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 213.735237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 213.918123] Lustre: lustre-MDT0000: new disk, initializing [ 214.099728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.128710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 220.336501] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.793197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 233.902770] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 233.952156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 233.957396] Lustre: Skipped 1 previous similar message [ 234.050150] Lustre: lustre-MDT0001: new disk, initializing [ 234.093609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 234.110597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 234.118364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 239.445176] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.687117] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 256.342729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.652862] Lustre: lustre-OST0000: new disk, initializing [ 256.655842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 256.661225] Lustre: 8455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.733560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.798405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 259.809587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 262.964711] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.216874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 277.936077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 278.166700] Lustre: lustre-OST0001: new disk, initializing [ 278.171209] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 278.180274] Lustre: 9528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 278.310522] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 284.232841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 284.240968] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 284.388775] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 285.388454] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 298.110366] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 306.159535] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 312.688182] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing check_logdir /tmp/testlogs/ [ 318.373259] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing yml_node [ 322.319837] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 325.172449] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 327.819418] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 329.604664] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 14 18:52:02 EDT 2026 [ 345.823564] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 347.420841] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 366.578031] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.591691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.603952] Lustre: Skipped 3 previous similar messages [ 366.620430] Lustre: Skipped 3 previous similar messages [ 371.681443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 371.696411] Lustre: Skipped 3 previous similar messages [ 376.802894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.810452] Lustre: Skipped 3 previous similar messages [ 377.312157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 377.515311] Lustre: server umount lustre-MDT0000 complete [ 381.377484] LustreError: 7484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747975 with bad export cookie 13512789439003417840 [ 381.385420] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.390222] LustreError: 7484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 381.932857] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 381.940643] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 381.946055] Lustre: Skipped 1 previous similar message [ 381.958667] Lustre: Skipped 1 previous similar message [ 387.040851] LustreError: 9773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 387.040961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 387.064413] LustreError: 9773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 387.995692] Lustre: server umount lustre-MDT0001 complete [ 392.410604] Lustre: server umount lustre-OST0000 complete [ 396.566506] Lustre: server umount lustre-OST0001 complete [ 403.617118] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 413.686687] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 451.997570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.251548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 452.276080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 452.348996] Lustre: lustre-MDT0000: new disk, initializing [ 452.416320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.431227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 457.911796] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 467.378577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.528880] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 467.584473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 467.587486] Lustre: Skipped 1 previous similar message [ 467.637965] Lustre: lustre-MDT0001: new disk, initializing [ 467.741144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 467.783143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 467.804089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 473.079865] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 481.756051] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 487.231497] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 494.743996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 495.075188] Lustre: lustre-OST0000: new disk, initializing [ 495.083343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 495.092885] Lustre: 17060:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 495.213906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.478870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 496.489785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 496.664334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 504.205899] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 516.338820] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 522.711180] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 523.342645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 530.600239] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 530.799767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 537.569227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 537.573990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.580807] LustreError: Skipped 1 previous similar message [ 537.581753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 537.581758] Lustre: Skipped 1 previous similar message [ 537.600512] Lustre: Skipped 1 previous similar message [ 547.299077] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 547.490550] Lustre: server umount lustre-OST0000 complete [ 552.932235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.945575] Lustre: Skipped 1 previous similar message [ 557.754736] Lustre: server umount lustre-MDT0000 complete [ 558.058116] LustreError: 15098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.085903] LustreError: 15098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 561.999961] LustreError: 17059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748156 with bad export cookie 13512789439003419422 [ 562.006821] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 562.016638] LustreError: 17059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 562.301910] Lustre: server umount lustre-MDT0001 complete [ 567.638342] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:56:00 (1786748160) [ 570.286788] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 572.085854] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:56:05 (1786748165) [ 578.317979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.819625] LustreError: 19470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.894666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 583.250491] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 584.177032] LustreError: 19471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 589.282379] LustreError: 19470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.233446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 597.400828] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 604.966879] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 611.539378] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 618.340207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 618.436890] LustreError: 21296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.423565] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 633.606826] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 644.582170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 644.587692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.599191] LustreError: Skipped 1 previous similar message [ 644.619642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 644.634855] Lustre: Skipped 1 previous similar message [ 644.670977] Lustre: Skipped 6 previous similar messages [ 648.294838] Lustre: server umount lustre-OST0000 complete [ 654.305797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 654.317200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 659.225622] Lustre: server umount lustre-MDT0000 complete [ 659.937518] LustreError: 19470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.952389] LustreError: 19470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 663.295353] LustreError: 19451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748257 with bad export cookie 13512789439003420346 [ 663.307086] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 663.312118] LustreError: 19451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 664.041350] Lustre: server umount lustre-MDT0001 complete [ 676.263582] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 680.713650] Key type lgssc unregistered [ 681.030330] LNet: 23009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 681.038475] LNetError: 23009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 682.086798] LNet: Removed LNI 192.168.202.118@tcp [ 683.165329] Key type .llcrypt unregistered [ 683.167444] Key type ._llcrypt unregistered [ 707.434189] Key type ._llcrypt registered [ 707.438170] Key type .llcrypt registered [ 707.605426] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 709.150263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 709.164016] alg: No test for adler32 (adler32-zlib) [ 710.441577] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 710.840436] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 712.528205] Key type lgssc registered [ 714.776303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 724.892326] loop: module loaded [ 724.983644] loop0: detected capacity change from 0 to 204800000 [ 725.463480] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 725.492078] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 726.045647] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.071660] loop0: detected capacity change from 0 to 204800000 [ 733.205655] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.210929] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.229796] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 738.132130] loop0: detected capacity change from 0 to 204800000 [ 744.835170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 744.862499] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 746.225611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 746.313101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 746.412296] Lustre: lustre-MDT0000: new disk, initializing [ 746.491397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.518407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 746.564352] Lustre: 25222:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 263, rollback = 2 [ 746.575276] Lustre: 25222:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 746.585434] Lustre: 25222:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/263/0 [ 746.593835] Lustre: 25222:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/2 [ 746.603864] Lustre: 25222:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 746.614078] Lustre: 25222:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 751.271874] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 760.284145] loop1: detected capacity change from 0 to 204800000 [ 766.157584] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 766.461709] Lustre: lustre-OST07c6: new disk, initializing [ 766.467293] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 766.474811] Lustre: Skipped 1 previous similar message [ 766.486281] Lustre: 26200:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 766.626550] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 773.750334] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 774.279500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 774.299646] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 774.359473] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 780.311658] Lustre: 25187:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 780.325217] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 780.333498] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 780.344752] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 780.353569] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 780.369037] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 804.976260] Lustre: 25187:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 804.987149] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 804.993482] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 805.002364] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 805.012445] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 805.019897] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 810.481570] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 810.499970] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 810.509566] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 810.518749] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 810.531312] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 810.545391] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 810.557167] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 810.569140] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 810.582770] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 810.593337] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 810.606448] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 810.619436] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 819.806479] Lustre: 25215:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 819.821659] Lustre: 25215:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 819.833075] Lustre: 25215:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 819.850650] Lustre: 25215:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 819.862037] Lustre: 25215:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 819.887195] Lustre: 25215:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 819.901792] Lustre: 25215:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 819.921450] Lustre: 25215:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 819.933324] Lustre: 25215:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 819.950644] Lustre: 25215:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 819.972589] Lustre: 25215:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 819.986660] Lustre: 25215:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 827.988108] Lustre: 25215:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 827.994143] Lustre: 25215:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 828.000042] Lustre: 25215:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 828.005535] Lustre: 25215:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 828.010232] Lustre: 25215:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 828.015761] Lustre: 25215:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 828.021529] Lustre: 25215:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 828.026193] Lustre: 25215:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 828.030620] Lustre: 25215:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 828.035402] Lustre: 25215:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 828.039731] Lustre: 25215:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 828.044352] Lustre: 25215:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 881.640808] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 881.696625] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 886.759037] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 887.328314] Lustre: server umount lustre-OST07c6 complete [ 898.709883] Lustre: server umount lustre-MDT0000 complete [ 916.351526] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 918.931949] Key type lgssc unregistered [ 919.278351] LNet: 28785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.290056] LNetError: 28785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.323588] LNet: Removed LNI 192.168.202.118@tcp [ 920.250647] Key type .llcrypt unregistered [ 920.255944] Key type ._llcrypt unregistered [ 939.193041] Key type ._llcrypt registered [ 939.194696] Key type .llcrypt registered [ 939.329221] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 940.607188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 940.651556] alg: No test for adler32 (adler32-zlib) [ 941.729247] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 941.898250] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 943.536201] Key type lgssc registered [ 944.715929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 953.863390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 953.878209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.435642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.443038] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 964.584058] LustreError: 30057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.604924] LustreError: 30057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 969.382276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 969.678047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 974.938879] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 983.556701] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 990.041472] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 997.353217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 997.654899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 999.728494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1004.243327] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1012.537198] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1030.635784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1030.639395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1030.657210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1030.662283] Lustre: Skipped 1 previous similar message [ 1033.199903] Lustre: server umount lustre-OST0000 complete [ 1040.866738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.880574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1040.895142] Lustre: Skipped 2 previous similar messages [ 1043.569219] Lustre: server umount lustre-MDT0000 complete [ 1045.998297] LustreError: 32502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.349761] LustreError: 30037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748642 with bad export cookie 16568275771060532942 [ 1048.356983] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1048.363167] LustreError: 30037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1048.762761] Lustre: server umount lustre-MDT0001 complete [ 1059.448040] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1062.188489] Key type lgssc unregistered [ 1062.461824] LNet: 33802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.473706] LNetError: 33802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.493703] LNet: Removed LNI 192.168.202.118@tcp [ 1063.219681] Key type .llcrypt unregistered [ 1063.222169] Key type ._llcrypt unregistered [ 1082.493942] Key type ._llcrypt registered [ 1082.496416] Key type .llcrypt registered [ 1082.563166] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1083.648676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1083.752705] alg: No test for adler32 (adler32-zlib) [ 1084.764582] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 1085.014575] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1086.712212] Key type lgssc registered [ 1088.439360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1096.762618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1096.776416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.248093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.039276] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1108.458832] LustreError: 35074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1108.470452] LustreError: 35074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1113.585310] LustreError: 35073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.432623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1114.817172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1121.436708] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1130.444989] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1137.542193] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1144.391454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1144.595096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.325373] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1159.167995] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1169.209736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1179.640800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.647821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1179.673321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1184.740299] LustreError: 37567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.757384] LustreError: 37567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1184.813990] Lustre: server umount lustre-OST0000 complete [ 1189.867156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.885256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1189.890487] Lustre: Skipped 2 previous similar messages [ 1189.901714] Lustre: Skipped 2 previous similar messages [ 1194.978618] LustreError: 36510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1194.996149] LustreError: 36510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1195.134871] Lustre: server umount lustre-MDT0000 complete [ 1199.696159] LustreError: 35055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748794 with bad export cookie 4739558267749565671 [ 1199.704991] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1199.706907] LustreError: 35055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1200.121651] Lustre: server umount lustre-MDT0001 complete [ 1210.824407] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1213.803990] Key type lgssc unregistered [ 1214.093883] LNet: 38637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1214.107493] LNetError: 38637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.127884] LNet: Removed LNI 192.168.202.118@tcp [ 1215.029693] Key type .llcrypt unregistered [ 1215.034980] Key type ._llcrypt unregistered [ 1240.692641] Key type ._llcrypt registered [ 1240.694618] Key type .llcrypt registered [ 1240.818791] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1241.927247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1242.077335] alg: No test for adler32 (adler32-zlib) [ 1243.763658] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 1244.174377] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1245.904168] Key type lgssc registered [ 1247.521791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1257.906117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1257.942369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.448491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.749432] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1268.707882] LustreError: 40054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.715536] LustreError: 40054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1273.775474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.830602] LustreError: 40055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.041319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1278.102103] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1286.874347] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1292.893508] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1299.285849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1299.791785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1303.882474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1306.574438] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1316.473314] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1329.301835] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 1329.638430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.654612] Lustre: Skipped 1 previous similar message [ 1333.595843] Lustre: server umount lustre-MDT0000 complete [ 1334.407754] LustreError: 40789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.500362] LustreError: 40035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748932 with bad export cookie 12563739054813919218 [ 1337.501486] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1337.511185] LustreError: 40035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1339.546960] LustreError: 40050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.572165] LustreError: 40050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1339.587546] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [ 1339.598660] Lustre: Skipped 3 previous similar messages [ 1339.876032] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.893141] Lustre: Skipped 1 previous similar message [ 1342.884146] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1342.889806] Lustre: Skipped 1 previous similar message [ 1343.956823] Lustre: server umount lustre-MDT0001 complete [ 1359.066995] Lustre: server umount lustre-OST0000 complete [ 1375.824852] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1378.505648] Key type lgssc unregistered [ 1378.798566] LNet: 43922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.816177] LNetError: 43922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.843404] LNet: Removed LNI 192.168.202.118@tcp [ 1379.673746] Key type .llcrypt unregistered [ 1379.675644] Key type ._llcrypt unregistered [ 1403.370454] Key type ._llcrypt registered [ 1403.374343] Key type .llcrypt registered [ 1403.551919] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1405.024151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1405.235778] alg: No test for adler32 (adler32-zlib) [ 1406.674243] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 1407.008914] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1408.736172] Key type lgssc registered [ 1410.429800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1423.368348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1423.396345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.589640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.668356] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1436.131791] LustreError: 45338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.154387] LustreError: 45338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1441.250781] LustreError: 45339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.262800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1441.720973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1446.587161] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1454.770827] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1461.259105] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1468.461733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1468.735578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.785445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1474.737430] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1482.415751] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1492.449689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.458976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1492.466647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1494.099300] Lustre: server umount lustre-OST0000 complete [ 1513.442186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1513.458541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.474718] Lustre: Skipped 1 previous similar message [ 1513.482239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1513.484727] Lustre: Skipped 1 previous similar message [ 1514.530884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.532039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.586718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.591784] Lustre: Skipped 2 previous similar messages [ 1524.710683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1524.712605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.264523] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1527.436996] Lustre: server umount lustre-MDT0000 complete [ 1529.828730] LustreError: 45333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1529.869096] LustreError: 45333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1532.631505] LustreError: 45319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749127 with bad export cookie 11097148951294422808 [ 1532.634151] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1532.660683] LustreError: 45319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1539.287310] Lustre: server umount lustre-MDT0001 complete [ 1553.389602] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1556.570529] Key type lgssc unregistered [ 1556.889703] LNet: 49054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.893607] LNetError: 49054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.915697] LNet: Removed LNI 192.168.202.118@tcp [ 1557.964901] Key type .llcrypt unregistered [ 1557.967947] Key type ._llcrypt unregistered [ 1581.210811] Key type ._llcrypt registered [ 1581.212875] Key type .llcrypt registered [ 1581.313770] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1582.749052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1582.838686] alg: No test for adler32 (adler32-zlib) [ 1583.864424] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 1584.075800] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1585.792188] Key type lgssc registered [ 1587.056537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1595.211783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1595.225420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.702409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1601.318553] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1607.146633] LustreError: 50467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.157819] LustreError: 50467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1609.137773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.513536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1613.499971] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1621.480639] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1628.428557] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1636.530371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1636.960152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.113574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1643.160507] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1650.863806] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1660.007278] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1664.117982] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1677.457269] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1419h14m28s [ 1679.744814] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1683.608612] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 1684.672365] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 1685.477470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.494642] Lustre: Skipped 2 previous similar messages [ 1686.565290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1686.567590] Lustre: Skipped 4 previous similar messages [ 1688.878953] Lustre: server umount lustre-MDT0000 complete [ 1690.751077] LustreError: 50463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1692.470949] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1692.650857] LustreError: 50466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1692.678809] LustreError: 50466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1696.907913] LustreError: 52853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1698.956280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.104736] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1699.318597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1699.368633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1700.334300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1703.431132] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1706.563658] Lustre: DEBUG MARKER: Wait for df (36725) ... [ 1708.478832] Lustre: DEBUG MARKER: done [ 1714.658243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.680496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.739193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1715.687956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1718.499286] Lustre: server umount lustre-OST0000 complete [ 1725.922740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1725.941327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.946687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1725.971327] Lustre: Skipped 1 previous similar message [ 1726.001236] Lustre: Skipped 3 previous similar messages [ 1736.673372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1736.677884] Lustre: Skipped 5 previous similar messages [ 1738.208154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1738.405178] Lustre: server umount lustre-MDT0000 complete [ 1741.732234] LustreError: 51195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.776287] LustreError: 51195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1743.514997] LustreError: 50448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749338 with bad export cookie 5165499150525291796 [ 1743.519962] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1743.522337] LustreError: 50448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1743.926031] Lustre: server umount lustre-MDT0001 complete [ 1758.421095] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1761.380109] Key type lgssc unregistered [ 1761.704568] LNet: 55400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.715324] LNetError: 55400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.736716] LNet: Removed LNI 192.168.202.118@tcp [ 1762.406596] Key type .llcrypt unregistered [ 1762.409603] Key type ._llcrypt unregistered [ 1806.182686] Key type ._llcrypt registered [ 1806.184804] Key type .llcrypt registered [ 1806.320574] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1807.749328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1807.906483] alg: No test for adler32 (adler32-zlib) [ 1809.055437] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 1809.377966] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1811.056170] Key type lgssc registered [ 1812.533885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1820.278263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1820.306455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.619736] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1821.657929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1821.863347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.695951] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1835.704805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.759948] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1835.789508] Lustre: 57607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1836.156183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1840.846784] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1849.361944] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1854.444943] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1861.140378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1861.325483] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1861.596973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1867.913299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1868.856150] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1878.068938] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1885.172177] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1419h17m58s [ 1886.958690] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1895.364825] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1898.034705] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1898.072810] LustreError: 57616:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1918.140088] Lustre: lustre-MDT0000: Client 7f6374a1-42cd-488e-b67a-9fa285e0a3df (at 192.168.202.18@tcp) reconnecting [ 1938.128399] LustreError: 57616:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1940.579456] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1942.684872] Lustre: DEBUG MARKER: done [ 1949.666655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1949.679330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.697779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1950.177125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1950.185305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.216339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1950.221585] Lustre: Skipped 1 previous similar message [ 1953.093792] Lustre: server umount lustre-OST0000 complete [ 1958.883678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1958.892282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.899702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1962.024262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1962.036910] Lustre: Skipped 4 previous similar messages [ 1963.379938] Lustre: server umount lustre-MDT0000 complete [ 1967.335728] LustreError: 57599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749561 with bad export cookie 12847482665235425801 [ 1967.347169] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1967.761612] Lustre: server umount lustre-MDT0001 complete [ 1978.520420] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1981.851653] Key type lgssc unregistered [ 1982.260837] LNet: 61941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.270948] LNetError: 61941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.296462] LNet: Removed LNI 192.168.202.118@tcp [ 1983.283526] Key type .llcrypt unregistered [ 1983.289585] Key type ._llcrypt unregistered [ 2031.198779] Key type ._llcrypt registered [ 2031.202275] Key type .llcrypt registered [ 2031.326951] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2032.245169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2032.373044] alg: No test for adler32 (adler32-zlib) [ 2033.385147] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 2033.628584] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2035.288174] Key type lgssc registered [ 2036.511531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2040.121968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2040.139235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.433486] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2041.455524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2041.607893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.413486] Lustre: Failing over lustre-MDT0000 [ 2042.653409] Lustre: server umount lustre-MDT0000 complete [ 2049.446982] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:20:42 (1786749642) [ 2056.615868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.289836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2062.375227] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2072.288149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.366629] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2072.387299] Lustre: 64744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2072.665829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2076.972933] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2085.004826] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2091.972861] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2101.074740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2101.194618] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2101.328480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.142471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2109.498621] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2119.177139] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2124.927740] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2134.003993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.012272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.013897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2134.029118] Lustre: Skipped 2 previous similar messages [ 2134.048976] Lustre: Skipped 2 previous similar messages [ 2137.938851] Lustre: server umount lustre-MDT0000 complete [ 2142.908907] LustreError: 66601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749737 with bad export cookie 10406341757670962406 [ 2142.915656] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2142.932686] LustreError: 66601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2143.350244] Lustre: server umount lustre-MDT0001 complete [ 2147.697651] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2152.714569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2160.352094] Lustre: 63665:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749738/real 1786749738] req@000000009c828a8a x1873542777972864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2160.379217] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.541471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.658283] Lustre: 63666:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749741/real 1786749741] req@00000000c824cc8e x1873542777972992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2164.768160] Lustre: 63666:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749743/real 1786749743] req@0000000027cba991 x1873542777973120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2168.219371] LustreError: 66595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.256176] LustreError: 66595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2168.334130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2168.450080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2169.313705] Lustre: 63664:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749747/real 1786749747] req@00000000d2e9145b x1873542777973376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2170.404483] LustreError: 66596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.414925] LustreError: 66596:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2173.396612] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2174.497929] Lustre: 63664:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749753/real 1786749753] req@000000008493838f x1873542777973504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2175.464662] LustreError: 68521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.478127] LustreError: 68521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2180.583670] LustreError: 68519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.612428] LustreError: 68519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2182.606710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2182.968381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2187.921890] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2188.272766] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2189.301485] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2197.733840] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2203.884992] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2213.870517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2213.877057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.897846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2219.244408] Lustre: server umount lustre-MDT0000 complete [ 2220.003569] LustreError: 68519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.036953] LustreError: 68519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2223.356763] LustreError: 64734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749817 with bad export cookie 10406341757670964688 [ 2223.371333] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2223.371978] LustreError: 64734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2223.908095] Lustre: server umount lustre-MDT0001 complete [ 2230.601636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2240.699312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.552299] Lustre: 63666:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749822/real 1786749822] req@0000000052b01205 x1873542778003712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749838 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2243.573513] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.580767] Lustre: Skipped 2 previous similar messages [ 2247.712738] LustreError: 63663:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000abfe451b x1873542778004480/t0(0) o250->MGC192.168.202.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2248.196248] LustreError: 66595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.217803] LustreError: 66595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2248.310592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.368918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2253.418542] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2253.428412] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2261.654888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.794414] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2267.634903] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2275.373858] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2282.336130] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2288.948709] LustreError: 72828:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (0000000067fd810d) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2289.073558] Lustre: server umount lustre-MDT0000 complete [ 2292.706743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2292.715703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.726159] LustreError: 66595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.735660] LustreError: 66595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2292.994506] LustreError: 67459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749887 with bad export cookie 10406341757670966067 [ 2293.012255] LustreError: 67459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2293.013958] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2293.338369] Lustre: server umount lustre-MDT0001 complete [ 2300.106183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2303.930839] Lustre: DEBUG MARKER: files compared the same [ 2306.330937] Lustre: server umount lustre-OST0000 complete [ 2329.345827] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2332.602619] Key type lgssc unregistered [ 2332.838518] LNet: 74483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2332.845163] LNetError: 74483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2332.859224] LNet: Removed LNI 192.168.202.118@tcp [ 2334.161471] Key type .llcrypt unregistered [ 2334.163606] Key type ._llcrypt unregistered [ 2364.565956] Key type ._llcrypt registered [ 2364.567786] Key type .llcrypt registered [ 2364.731104] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2366.476382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.587183] alg: No test for adler32 (adler32-zlib) [ 2367.869292] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 2368.245296] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2370.024160] Key type lgssc registered [ 2371.190420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2380.767353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2380.780314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.327961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2388.229654] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug +malloc all [ 2391.520935] LustreError: 75900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.549658] LustreError: 75900:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2396.645899] LustreError: 75901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.084885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.471108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2403.780103] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug +malloc all [ 2412.911773] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2419.990686] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2427.763678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2428.148054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2435.734662] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug +malloc all [ 2444.308893] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2451.711810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2462.176896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2462.182594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2462.189271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2462.192614] Lustre: Skipped 1 previous similar message [ 2467.300644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2467.308294] Lustre: Skipped 1 previous similar message [ 2468.390409] Lustre: server umount lustre-OST0000 complete [ 2475.489211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2475.502182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2475.534967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2475.540436] Lustre: Skipped 1 previous similar message [ 2477.536990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2477.537112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.513228] Lustre: server umount lustre-MDT0000 complete [ 2482.658321] LustreError: 75895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.698911] LustreError: 75895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2484.837559] LustreError: 78414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750079 with bad export cookie 5022616956372043910 [ 2484.839746] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2484.848936] LustreError: 78414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2485.090066] Lustre: server umount lustre-MDT0001 complete [ 2498.767488] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2501.939648] Key type lgssc unregistered [ 2502.268658] LNet: 79470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2502.284750] LNetError: 79470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2503.346455] LNet: Removed LNI 192.168.202.118@tcp [ 2504.248646] Key type .llcrypt unregistered [ 2504.250891] Key type ._llcrypt unregistered [ 2532.003345] Key type ._llcrypt registered [ 2532.005451] Key type .llcrypt registered [ 2532.102765] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2533.386134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2533.442271] alg: No test for adler32 (adler32-zlib) [ 2534.635281] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 2534.987554] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2536.712264] Key type lgssc registered [ 2537.996878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2546.753807] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2546.765912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2562.339042] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2567.458899] LustreError: 80861:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.118@tcp: failed processing log, type 4: rc = -110 [ 2594.208976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2601.733275] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2608.519379] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2616.911215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.051186] Lustre: *** cfs_fail_loc=706, val=0*** [ 2617.496644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.574271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2622.946355] LustreError: 80887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.969883] LustreError: 80887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2623.416591] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2628.081209] LustreError: 80886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.098238] LustreError: 81947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.135787] LustreError: 81947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2634.217061] LustreError: 81947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.256203] LustreError: 81947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2634.791981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.129781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2640.841832] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2648.215625] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2655.496971] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2660.849281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.873480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2660.899372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2660.923333] Lustre: Skipped 1 previous similar message [ 2665.955025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2671.076209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2671.085138] Lustre: Skipped 2 previous similar messages [ 2673.120757] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2673.376526] Lustre: server umount lustre-OST0000 complete [ 2681.314981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.316118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2681.333975] Lustre: Skipped 1 previous similar message [ 2681.338966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2681.375871] Lustre: Skipped 1 previous similar message [ 2684.171908] Lustre: server umount lustre-MDT0000 complete [ 2686.441032] LustreError: 81951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.476215] LustreError: 81951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2688.950032] LustreError: 80868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750283 with bad export cookie 5525676880191512362 [ 2688.953663] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2688.959668] LustreError: 80868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2689.182350] Lustre: server umount lustre-MDT0001 complete [ 2703.488877] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2706.740103] Key type lgssc unregistered [ 2707.006965] LNet: 84487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2707.014043] LNetError: 84487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2707.030248] LNet: Removed LNI 192.168.202.118@tcp [ 2707.882866] Key type .llcrypt unregistered [ 2707.886362] Key type ._llcrypt unregistered [ 2731.906428] Key type ._llcrypt registered [ 2731.908182] Key type .llcrypt registered [ 2732.002788] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2732.908351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2732.998489] alg: No test for adler32 (adler32-zlib) [ 2734.052141] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 2734.389427] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2736.072222] Key type lgssc registered [ 2737.187639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2745.417481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2745.433692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.408618] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2759.261029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.910918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2764.371845] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2765.295030] LustreError: 86572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.336777] LustreError: 86572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2770.416631] LustreError: 86570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.442367] LustreError: 86570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2773.607556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.967519] LustreError: 86575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.996569] LustreError: 86575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2774.007096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.114554] LustreError: 86576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.147090] LustreError: 86576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2782.270084] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2784.226799] LustreError: 86575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2784.252414] LustreError: 86575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2791.406447] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2791.432760] Lustre: MGS: Client 55d45af9-c7fc-4c29-addf-14d71dd84d96 (at 0@lo) reconnecting [ 2791.462309] LustreError: 86586:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2791.698891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.781907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2796.207890] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2812.387834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2812.389338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.414086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2812.453371] Lustre: Skipped 1 previous similar message [ 2815.695976] Lustre: server umount lustre-OST0000 complete [ 2822.625033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2822.629887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.645678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2822.655462] Lustre: Skipped 1 previous similar message [ 2822.669662] Lustre: Skipped 2 previous similar messages [ 2825.888606] Lustre: server umount lustre-MDT0000 complete [ 2827.748602] LustreError: 86572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.772801] LustreError: 86572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2830.466927] Lustre: server umount lustre-MDT0001 complete [ 2834.621435] Lustre: server umount MGS complete [ 2847.275904] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:34:00 (1786750440) [ 2879.992376] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 2889.608939] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2928.657162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2934.118527] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2944.046549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.320027] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2944.375801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2944.464827] Lustre: lustre-MDT0001: new disk, initializing [ 2944.519053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2944.608284] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2944.642359] Lustre: 93743:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2944.654431] LustreError: 93743:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2944.684961] LustreError: 93745:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2944.710846] LustreError: 93745:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2945.185900] LustreError: 93745:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2945.201852] LustreError: 93745:0:(fld_handler.c:275:fld_server_lookup()) Skipped 2026 previous similar messages [ 2945.211706] LustreError: 93745:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2945.223467] LustreError: 93745:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) Skipped 2039 previous similar messages [ 2946.185854] LustreError: 93745:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2946.191628] LustreError: 93745:0:(fld_handler.c:275:fld_server_lookup()) Skipped 4558 previous similar messages [ 2946.212394] LustreError: 93745:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2946.224632] LustreError: 93745:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) Skipped 4679 previous similar messages [ 2948.186342] LustreError: 93745:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2948.193943] LustreError: 93745:0:(fld_handler.c:275:fld_server_lookup()) Skipped 11872 previous similar messages [ 2948.245368] LustreError: 93745:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2948.257218] LustreError: 93745:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) Skipped 11839 previous similar messages [ 2951.052444] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2952.187861] LustreError: 93745:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2952.197748] LustreError: 93745:0:(fld_handler.c:275:fld_server_lookup()) Skipped 15511 previous similar messages [ 2952.240822] LustreError: 93745:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2952.245035] LustreError: 93745:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) Skipped 15515 previous similar messages [ 2960.190278] LustreError: 93745:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2960.245246] LustreError: 93745:0:(fld_handler.c:275:fld_server_lookup()) Skipped 32342 previous similar messages [ 2960.249326] LustreError: 93745:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2960.252894] LustreError: 93745:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) Skipped 32237 previous similar messages [ 2966.500808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2966.900229] Lustre: lustre-OST0000: new disk, initializing [ 2966.910141] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2966.919503] Lustre: 94511:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2967.089809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.814833] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2976.189092] LustreError: 93745:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2976.193865] LustreError: 93745:0:(fld_handler.c:275:fld_server_lookup()) Skipped 83891 previous similar messages [ 2976.249072] LustreError: 93745:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2976.263089] LustreError: 93745:0:(mdt_handler.c:8434:mdt_trash_setup_thread()) Skipped 84486 previous similar messages [ 2977.888839] Lustre: 94960:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000b7048b05 x1873543511977472/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786750588 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2987.135728] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2998.258434] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2998.479906] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2998.503508] Lustre: MGS: Client dbb3400f-c55d-4596-8b0e-dd2b85109d1c (at 0@lo) reconnecting [ 2998.536828] LustreError: 93729:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2998.604727] Lustre: 93011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2998.734955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2998.930703] Lustre: lustre-MDT0000: new disk, initializing [ 2999.222479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.290071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3000.373085] Lustre: 95581:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000408308d2 x1873543511985152/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786750610 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3004.659416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3004.667358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3005.421138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3005.606795] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3005.688111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3035.622709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3035.626533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3035.630165] Lustre: Skipped 1 previous similar message [ 3037.751338] Lustre: server umount lustre-OST0000 complete [ 3045.858587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3045.859411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.877333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3045.890642] Lustre: Skipped 1 previous similar message [ 3045.909312] Lustre: Skipped 2 previous similar messages [ 3047.880413] Lustre: server umount lustre-MDT0000 complete [ 3050.980185] LustreError: 93718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3050.997321] LustreError: 93718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3066.336851] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3066.679936] Lustre: server umount lustre-MDT0001 complete [ 3070.583284] Lustre: server umount MGS complete [ 3087.353359] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:37:59 (1786750679) [ 3109.406735] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3113.881194] Key type lgssc unregistered [ 3114.359633] LNet: 98087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3114.376276] LNetError: 98087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3114.397505] LNet: Removed LNI 192.168.202.118@tcp [ 3115.081886] Key type .llcrypt unregistered [ 3115.082961] Key type ._llcrypt unregistered [ 3132.373142] Key type ._llcrypt registered [ 3132.376716] Key type .llcrypt registered [ 3132.525384] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3133.748240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3133.792834] alg: No test for adler32 (adler32-zlib) [ 3134.725453] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 3134.925796] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3136.536208] Key type lgssc registered [ 3137.923896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3148.407757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3148.451381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3150.023396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3154.789808] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3159.522961] LustreError: 99309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.543654] LustreError: 99309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3163.864581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.218772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3169.528117] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3177.530795] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3183.603283] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3191.543318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3191.915605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.000057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3199.060648] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3210.245268] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3219.948138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.965295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3219.972088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3220.003638] Lustre: Skipped 1 previous similar message [ 3225.058476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3225.068892] Lustre: Skipped 1 previous similar message [ 3225.372225] Lustre: server umount lustre-OST0000 complete [ 3230.178708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.187193] Lustre: Skipped 1 previous similar message [ 3230.191246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.195858] Lustre: Skipped 1 previous similar message [ 3235.300139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3235.310817] Lustre: Skipped 1 previous similar message [ 3236.012653] Lustre: server umount lustre-MDT0000 complete [ 3240.421269] LustreError: 100850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.436766] LustreError: 100850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3240.485660] LustreError: 99292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750835 with bad export cookie 15725275810465983397 [ 3240.499487] LustreError: 99292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3240.501378] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3240.767594] Lustre: server umount lustre-MDT0001 complete [ 3252.659248] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3255.689827] Key type lgssc unregistered [ 3255.924274] LNet: 102850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3255.931925] LNetError: 102850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3255.947339] LNet: Removed LNI 192.168.202.118@tcp [ 3256.803214] Key type .llcrypt unregistered [ 3256.805448] Key type ._llcrypt unregistered [ 3275.570952] Key type ._llcrypt registered [ 3275.572536] Key type .llcrypt registered [ 3275.654710] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3276.634416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3276.805195] alg: No test for adler32 (adler32-zlib) [ 3278.001773] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 3278.338180] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3280.032240] Key type lgssc registered [ 3281.204681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3287.921384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3287.932753] LustreError: 103931:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3287.938179] LustreError: 103933:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3287.947334] LustreError: 103931:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 3287.954775] LustreError: 103931:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3287.960917] LustreError: 103931:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3287.962556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3289.551365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3298.368467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.617774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3303.821087] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3310.026937] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing lsmod [ 3313.116986] LustreError: 105144:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3313.133273] LustreError: 105143:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3313.139457] LustreError: 105144:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 3313.145847] LustreError: 105144:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3313.158921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3313.159328] LustreError: 105144:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3313.550321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.613867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [ 3319.268706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3319.279965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3319.299432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3322.699678] Lustre: server umount lustre-MDT0000 complete [ 3323.878766] LustreError: 103970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3323.890184] LustreError: 103970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3326.344524] LustreError: 103946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750920 with bad export cookie 13567558208179541535 [ 3326.346177] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3326.354502] LustreError: 103946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3328.032492] LustreError: 104404:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000adb8e1d8 x1873544082380416/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3328.248828] Lustre: server umount lustre-MDT0001 complete [ 3342.548828] Lustre: server umount lustre-OST0000 complete [ 3350.977947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.708798] LustreError: 106208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3351.777404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3355.973819] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3357.154890] LustreError: 106209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.276026] LustreError: 106209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3364.356448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3364.726273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3368.531768] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3376.998481] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3383.601961] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3392.577504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3392.830851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.916457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:97) [ 3399.208281] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3407.882140] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3426.575915] Lustre: server umount lustre-OST0000 complete [ 3431.905221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3431.916507] Lustre: Skipped 2 previous similar messages [ 3431.921957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3431.935616] Lustre: Skipped 2 previous similar messages [ 3436.524601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3436.536238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3436.552548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3442.145408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3442.154818] Lustre: Skipped 3 previous similar messages [ 3442.912224] Lustre: 103336:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751021/real 1786751021] req@000000001f07ea77 x1873544082411520/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786751037 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3442.972283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3446.240160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3446.720637] Lustre: server umount lustre-MDT0000 complete [ 3448.290959] Lustre: 103335:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751026/real 1786751026] req@0000000070e197ad x1873544082413568/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786751042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3448.297620] LustreError: 107268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.363880] LustreError: 107268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3450.880548] LustreError: 106190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751045 with bad export cookie 13567558208179542480 [ 3450.894771] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3450.897436] LustreError: 106190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3451.370738] Lustre: server umount lustre-MDT0001 complete [ 3461.698531] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3464.416361] Key type lgssc unregistered [ 3464.741852] LNet: 109768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3464.750753] LNetError: 109768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3464.765925] LNet: Removed LNI 192.168.202.118@tcp [ 3465.695232] Key type .llcrypt unregistered [ 3465.700266] Key type ._llcrypt unregistered [ 3490.605704] Key type ._llcrypt registered [ 3490.607752] Key type .llcrypt registered [ 3490.731933] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3492.173210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3492.219827] alg: No test for adler32 (adler32-zlib) [ 3493.346572] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 3493.654968] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3495.312124] Key type lgssc registered [ 3496.793875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3506.681730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3506.696983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3508.329101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3512.595729] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3517.924697] LustreError: 111187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3517.978031] LustreError: 111187:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3523.060726] LustreError: 111186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.243825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.669626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3529.883679] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3538.547775] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3545.196408] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3552.845226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3553.240704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3558.404507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3559.761296] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3568.050126] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3573.045548] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3575.726206] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3583.975456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3583.984263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.986094] LustreError: Skipped 1 previous similar message [ 3583.993691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3589.093780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3589.099614] Lustre: Skipped 2 previous similar messages [ 3594.211118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3594.217463] Lustre: Skipped 1 previous similar message [ 3595.232283] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3595.437652] Lustre: server umount lustre-OST0000 complete [ 3599.332404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3599.338725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3599.345447] Lustre: Skipped 2 previous similar messages [ 3599.356280] Lustre: Skipped 1 previous similar message [ 3604.451621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3604.459025] Lustre: Skipped 1 previous similar message [ 3605.808763] Lustre: server umount lustre-MDT0000 complete [ 3609.569917] LustreError: 113733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3609.579256] LustreError: 113733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3609.649879] LustreError: 111168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751204 with bad export cookie 2343447518413093009 [ 3609.658941] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3609.663937] LustreError: 111168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3609.854714] Lustre: server umount lustre-MDT0001 complete [ 3620.935301] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3623.880634] Key type lgssc unregistered [ 3624.132114] LNet: 114832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3624.136053] LNetError: 114832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3624.145161] LNet: Removed LNI 192.168.202.118@tcp [ 3624.608464] Key type .llcrypt unregistered [ 3624.611051] Key type ._llcrypt unregistered [ 3642.529210] Key type ._llcrypt registered [ 3642.538497] Key type .llcrypt registered [ 3642.702368] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3644.013831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3644.043303] alg: No test for adler32 (adler32-zlib) [ 3645.751683] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 3646.123383] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3647.864502] Key type lgssc registered [ 3649.240618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3656.963212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3656.987508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.823692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3664.767810] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3668.970022] LustreError: 116054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3668.989719] LustreError: 116054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3674.085393] LustreError: 116055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3674.304734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3674.811316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3679.527458] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3688.000641] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3694.133787] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3701.439287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3701.896481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3703.925719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3710.396536] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3719.194592] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3730.923187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.927229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3730.935567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3730.959336] Lustre: Skipped 1 previous similar message [ 3730.980570] Lustre: Skipped 1 previous similar message [ 3736.042938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.341592] Lustre: server umount lustre-OST0000 complete [ 3741.164657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3741.170485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3741.173462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3741.173470] Lustre: Skipped 1 previous similar message [ 3741.212833] Lustre: Skipped 1 previous similar message [ 3746.278433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3746.294696] Lustre: Skipped 2 previous similar messages [ 3747.414880] Lustre: server umount lustre-MDT0000 complete [ 3751.394931] LustreError: 116054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3751.414664] LustreError: 116054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3752.539593] LustreError: 116035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751347 with bad export cookie 12673594518936667450 [ 3752.552213] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3752.552979] LustreError: 116035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3752.878898] Lustre: server umount lustre-MDT0001 complete [ 3767.111964] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3771.099154] Key type lgssc unregistered [ 3771.320783] LNet: 119596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3771.326158] LNetError: 119596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3771.350344] LNet: Removed LNI 192.168.202.118@tcp [ 3772.969663] Key type .llcrypt unregistered [ 3772.973404] Key type ._llcrypt unregistered [ 3798.312784] Key type ._llcrypt registered [ 3798.315277] Key type .llcrypt registered [ 3798.435345] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3800.054951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3800.103525] alg: No test for adler32 (adler32-zlib) [ 3801.384896] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 3801.649208] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3803.400242] Key type lgssc registered [ 3804.841826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3813.816440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3813.837231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.437712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3820.377036] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3824.612449] LustreError: 121014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3824.638472] LustreError: 121014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3829.264339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.733438] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3830.051711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3835.490338] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3844.766841] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3851.323384] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3858.050583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3858.371555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.412945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3865.082046] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3873.761863] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3881.525493] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3885.104593] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3885.109880] Lustre: Skipped 2 previous similar messages [ 3888.818556] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3888.878112] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3889.833360] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3889.843690] Lustre: Skipped 2 previous similar messages [ 3896.900477] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3896.909626] Lustre: Skipped 1 previous similar message [ 3901.485357] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.18@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3901.494597] Lustre: Skipped 2 previous similar messages [ 3903.625957] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.18@tcp 192.168.0.[2,10]@tcp [ 3907.952901] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.18/24@tcp in log lustre-MDT0000 [ 3907.963942] Lustre: Skipped 2 previous similar messages [ 3912.846172] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.18/24@tcp [ 3912.858886] Lustre: Skipped 1 previous similar message [ 3918.824354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3918.827492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3918.833572] LustreError: Skipped 2 previous similar messages [ 3918.855886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3920.220843] Lustre: server umount lustre-OST0000 complete [ 3925.986533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.022123] Lustre: Skipped 1 previous similar message [ 3926.022961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3926.046193] Lustre: Skipped 1 previous similar message [ 3929.941721] Lustre: server umount lustre-MDT0000 complete [ 3931.108584] LustreError: 123832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3931.143700] LustreError: 123832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3933.788158] LustreError: 123648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751528 with bad export cookie 9120163812692270604 [ 3933.794671] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3933.795743] LustreError: 123648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3944.708588] Lustre: server umount lustre-MDT0001 complete [ 3957.973479] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3961.258865] Key type lgssc unregistered [ 3961.587689] LNet: 125870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3961.592595] LNetError: 125870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3961.630181] LNet: Removed LNI 192.168.202.118@tcp [ 3962.548336] Key type .llcrypt unregistered [ 3962.551364] Key type ._llcrypt unregistered [ 3992.854732] Key type ._llcrypt registered [ 3992.858481] Key type .llcrypt registered [ 3992.993687] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3994.323811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3994.363846] alg: No test for adler32 (adler32-zlib) [ 3995.509864] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 3995.784066] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3997.480240] Key type lgssc registered [ 3998.664616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4006.892245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4006.923351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4008.401872] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4008.423737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4008.432566] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.18/24@tcp [ 4012.607447] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4017.637465] LustreError: 127434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4017.652714] LustreError: 127434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4021.625307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.059091] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4022.091535] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.18/24@tcp [ 4022.119922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4026.965691] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4034.350382] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4040.989493] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4047.936150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4048.317203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4053.461681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:201 to 0x280000401:225) [ 4054.527922] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4062.265729] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4074.980451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4074.993977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4075.026468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4077.029155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4077.046820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4077.049988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4077.079855] Lustre: Skipped 1 previous similar message [ 4082.150386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4082.157180] Lustre: Skipped 1 previous similar message [ 4087.266364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4088.288679] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4088.471876] Lustre: server umount lustre-OST0000 complete [ 4093.933215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4093.955108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.976621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.982790] Lustre: Skipped 1 previous similar message [ 4097.514559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.944445] Lustre: server umount lustre-MDT0000 complete [ 4102.570971] LustreError: 128177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751697 with bad export cookie 12854652820016284664 [ 4102.573469] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4102.586518] LustreError: 128177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4102.824617] Lustre: server umount lustre-MDT0001 complete [ 4114.572750] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 4117.588489] Key type lgssc unregistered [ 4117.892595] LNet: 131027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4117.909529] LNetError: 131027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4117.928251] LNet: Removed LNI 192.168.202.118@tcp [ 4118.773167] Key type .llcrypt unregistered [ 4118.775377] Key type ._llcrypt unregistered [ 4146.722591] Key type ._llcrypt registered [ 4146.724582] Key type .llcrypt registered [ 4146.794540] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 4165.043815] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4165.997675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4166.042463] alg: No test for adler32 (adler32-zlib) [ 4167.517083] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 4167.765479] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 4169.408661] Key type lgssc registered [ 4170.707823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4208.109026] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4208.123721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4209.333886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4209.357652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4209.426191] Lustre: lustre-MDT0000: new disk, initializing [ 4209.488208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4209.501989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4213.864996] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4225.045330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.321298] Lustre: 134425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4225.410684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4225.413449] Lustre: Skipped 1 previous similar message [ 4225.467812] Lustre: lustre-MDT0001: new disk, initializing [ 4225.556562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4225.637125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4225.655073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4233.036787] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4243.862246] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4250.613266] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4258.207680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4258.620078] Lustre: lustre-OST0000: new disk, initializing [ 4258.628828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4258.641261] Lustre: 136399:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4258.775878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.429186] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4266.612652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4266.628250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4266.810415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4276.681850] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4281.797682] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4282.160408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4287.641243] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4287.936904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4292.580488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4292.583204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.587525] LustreError: Skipped 1 previous similar message [ 4292.587996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4292.593099] Lustre: Skipped 1 previous similar message [ 4295.745989] Lustre: server umount lustre-OST0000 complete [ 4302.310810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4302.325682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4302.343382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4302.348688] Lustre: Skipped 1 previous similar message [ 4306.129286] Lustre: server umount lustre-MDT0000 complete [ 4310.504671] LustreError: 135222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751905 with bad export cookie 5440733126933802806 [ 4310.507316] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4310.528320] LustreError: 135222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4310.837051] Lustre: server umount lustre-MDT0001 complete [ 4321.315614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4321.915651] LustreError: 138616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4321.957241] LustreError: 138616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4322.033580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4327.396460] LustreError: 138617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4327.424501] LustreError: 138617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4328.838907] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4331.491034] LustreError: 138616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4336.612055] LustreError: 138617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4340.659170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.103769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4345.782521] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4354.704607] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4361.943265] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4369.725989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4370.020489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4376.544529] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4385.035410] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4390.241842] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4392.698847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4397.953707] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4398.313604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4408.385650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4408.518902] Lustre: lustre-OST0001: new disk, initializing [ 4408.521246] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4408.525762] Lustre: 142009:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4408.630953] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4415.112480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4415.126374] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4415.264220] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4416.173068] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4424.351427] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4424.525964] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4429.600060] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4429.981622] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4436.715502] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff995f908fe800.ost_server_uuid 50 [ 4438.389855] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff995f908fe800.ost_server_uuid in IDLE state after 0 sec [ 4446.190570] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4446.193803] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.207663] LustreError: Skipped 1 previous similar message [ 4446.220556] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4446.229437] Lustre: Skipped 2 previous similar messages [ 4446.256052] Lustre: Skipped 2 previous similar messages [ 4451.231904] Lustre: server umount lustre-OST0001 complete [ 4451.305913] LustreError: 141566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.321812] LustreError: 141566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4456.423214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4456.449483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4456.466975] Lustre: Skipped 1 previous similar message [ 4461.538674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4461.540783] LustreError: 140446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4461.560980] Lustre: Skipped 1 previous similar message [ 4461.600815] LustreError: 140446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4462.524334] Lustre: server umount lustre-OST0000 complete [ 4469.216845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4469.221770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.231080] Lustre: Skipped 1 previous similar message [ 4471.789104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4471.791785] Lustre: Skipped 1 previous similar message [ 4476.451337] Lustre: server umount lustre-MDT0000 complete [ 4480.244336] LustreError: 138597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752074 with bad export cookie 5440733126933803730 [ 4480.252305] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4480.257307] LustreError: 138597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4480.607676] Lustre: server umount lustre-MDT0001 complete [ 4500.395815] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 4503.325713] Key type lgssc unregistered [ 4503.563643] LNet: 145027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4503.567590] LNetError: 145027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4503.580488] LNet: Removed LNI 192.168.202.118@tcp [ 4504.562809] Key type .llcrypt unregistered [ 4504.566619] Key type ._llcrypt unregistered [ 4547.336807] Key type ._llcrypt registered [ 4547.342842] Key type .llcrypt registered [ 4547.542470] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 4567.769841] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4569.590859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4569.656810] alg: No test for adler32 (adler32-zlib) [ 4570.685592] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 4570.925703] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 4572.576151] Key type lgssc registered [ 4573.918951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4617.109329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4617.129838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4618.459712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4618.508132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4618.628510] Lustre: lustre-MDT0000: new disk, initializing [ 4618.719709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4618.745715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4624.206520] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4637.203810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4637.507935] Lustre: 149189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4637.660843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4637.670338] Lustre: Skipped 1 previous similar message [ 4637.858645] Lustre: lustre-MDT0001: new disk, initializing [ 4638.063083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4638.137254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4638.158518] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4642.815255] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4654.481673] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4661.085656] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4668.386025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4668.563190] Lustre: lustre-OST0000: new disk, initializing [ 4668.567614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4668.571923] Lustre: 151164:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4668.667632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.527146] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4678.795643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4678.815037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4678.980948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4689.142546] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4694.241414] Lustre: Failing over lustre-OST0000 [ 4694.509692] Lustre: server umount lustre-OST0000 complete [ 4710.900512] Lustre: 147412:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752289/real 1786752289] req@000000006dd89724 x1873545438204672/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786752305 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4710.911197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4710.941688] Lustre: 147412:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4710.997229] Lustre: Skipped 1 previous similar message [ 4712.034383] Lustre: 147414:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752290/real 1786752290] req@00000000729769a1 x1873545438205440/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786752306 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4715.304481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4715.883840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4715.915715] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4716.901750] Lustre: 147413:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752294/real 1786752294] req@000000009ac97b58 x1873545438206464/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786752310 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4721.007095] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4721.856443] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4721.859433] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4722.912311] Lustre: 147411:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752299/real 1786752299] req@00000000b0691370 x1873545438207744/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786752315 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4722.962429] Lustre: 147411:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4723.773408] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4727.980897] Lustre: Failing over lustre-MDT0000 [ 4728.320622] Lustre: server umount lustre-MDT0000 complete [ 4730.337114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4730.359239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4731.370385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4731.390557] Lustre: Skipped 1 previous similar message [ 4736.482177] LustreError: 150357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4736.496989] LustreError: 150357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4737.153936] LustreError: 149196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.602429] LustreError: 149198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.628136] LustreError: 149198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4746.723150] Lustre: 147414:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752325/real 1786752325] req@000000008c9320ac x1873545438220928/t0(0) o400->MGC192.168.202.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786752341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4746.733884] LustreError: 150356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4746.790186] Lustre: 147414:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4746.790299] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4746.845057] LustreError: 150356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4748.680332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.842544] LustreError: 149202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.869491] LustreError: 149202:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4757.551035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4757.620451] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4757.640544] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4762.537878] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4762.617848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4762.645273] Lustre: Skipped 1 previous similar message [ 4762.761077] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4762.851755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4772.839286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4772.841988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4772.852835] Lustre: Skipped 1 previous similar message [ 4777.963751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4777.971571] Lustre: Skipped 1 previous similar message [ 4778.137531] Lustre: server umount lustre-OST0000 complete [ 4783.075794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4783.115859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4783.127138] Lustre: Skipped 2 previous similar messages [ 4788.195082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4788.203836] Lustre: Skipped 1 previous similar message [ 4788.809484] Lustre: server umount lustre-MDT0000 complete [ 4793.240960] LustreError: 149182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752387 with bad export cookie 11463811662243638201 [ 4793.252359] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4793.263746] LustreError: 149182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4793.557973] Lustre: server umount lustre-MDT0001 complete [ 4805.650392] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 4809.120741] Key type lgssc unregistered [ 4809.497978] LNet: 154825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4809.515386] LNetError: 154825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4809.530029] LNet: Removed LNI 192.168.202.118@tcp [ 4810.388905] Key type .llcrypt unregistered [ 4810.391636] Key type ._llcrypt unregistered [ 4853.147894] Key type ._llcrypt registered [ 4853.150869] Key type .llcrypt registered [ 4853.273312] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4854.899243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4854.972448] alg: No test for adler32 (adler32-zlib) [ 4856.179499] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 4856.474444] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 4858.216767] Key type lgssc registered [ 4859.448843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4868.418268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4868.432699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4869.734656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4873.650286] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4879.859160] LustreError: 156349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4879.882447] LustreError: 156349:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4885.002033] LustreError: 156350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4885.923874] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4890.091627] LustreError: 156349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4890.295302] LustreError: 156378:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4890.306838] LustreError: 156378:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4890.312057] LustreError: 156378:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4890.761813] Lustre: server umount lustre-MDT0000 complete [ 4909.353977] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 4919.607788] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4960.732754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4961.089129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4961.116924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4961.189550] Lustre: lustre-MDT0000: new disk, initializing [ 4961.254103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4961.297371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4966.795754] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4979.432084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4979.570939] Lustre: 159686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4979.638946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4979.643400] Lustre: Skipped 1 previous similar message [ 4979.744287] Lustre: lustre-MDT0001: new disk, initializing [ 4979.899964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4979.943455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4979.956932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4984.723514] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4996.371953] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5001.810198] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5008.482096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5008.719983] Lustre: lustre-OST0000: new disk, initializing [ 5008.722733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5008.729708] Lustre: 161656:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5008.806694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5014.147062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5014.168972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5014.356112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5015.003439] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5024.541660] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5029.720544] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5031.671981] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:10:24 (1786752624) [ 5297.633220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5297.644932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.702504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5301.218256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5301.220153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5301.233474] Lustre: Skipped 1 previous similar message [ 5301.259452] Lustre: Skipped 2 previous similar messages [ 5303.718695] Lustre: server umount lustre-MDT0000 complete [ 5306.383488] LustreError: 159697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5306.400062] LustreError: 159697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5308.344283] LustreError: 163424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752902 with bad export cookie 10693166185022087198 [ 5308.348329] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5308.354175] LustreError: 163424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5311.464750] LustreError: 161948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5311.465685] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5311.486342] LustreError: 161948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5311.493333] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5313.504885] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5314.720787] Lustre: server umount lustre-MDT0001 complete [ 5325.316121] Lustre: server umount lustre-OST0000 complete [ 5335.935662] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 5347.516322] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5390.239575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5390.544794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5390.611566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5390.782219] Lustre: lustre-MDT0000: new disk, initializing [ 5390.936697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5390.976963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5397.954424] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5411.622349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.817754] Lustre: 166324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5411.925951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5411.930247] Lustre: Skipped 1 previous similar message [ 5412.038719] Lustre: lustre-MDT0001: new disk, initializing [ 5412.188779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5412.237155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5412.251917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5418.820918] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5429.864563] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5436.163376] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5444.169909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5444.518374] Lustre: lustre-OST0000: new disk, initializing [ 5444.523301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5444.546583] Lustre: 168293:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5444.656520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5451.382401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5451.399956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5451.553300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5452.323272] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5464.291123] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5469.913446] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5470.111835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5475.496663] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5475.707996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5482.468792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5482.472061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5482.488716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5482.496751] Lustre: Skipped 1 previous similar message [ 5484.021487] Lustre: server umount lustre-OST0000 complete [ 5489.122261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5489.139092] LustreError: Skipped 1 previous similar message [ 5489.152536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5489.192168] Lustre: Skipped 1 previous similar message [ 5492.712880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5492.724118] Lustre: Skipped 1 previous similar message [ 5495.009986] Lustre: server umount lustre-MDT0000 complete [ 5497.833985] LustreError: 166336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5497.847254] LustreError: 166336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5500.413418] LustreError: 166315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753094 with bad export cookie 10693166185022246070 [ 5500.434253] LustreError: 166315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5500.434758] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5500.952179] Lustre: server umount lustre-MDT0001 complete [ 5512.159545] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:18:24 (1786753104) [ 5531.574406] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 5544.629734] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5587.797800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5588.031457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5588.065839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5588.147828] Lustre: lustre-MDT0000: new disk, initializing [ 5588.215537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5588.230211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5594.097405] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5607.203025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5607.367964] Lustre: 173070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5607.673082] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5611.784739] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5622.017457] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5628.673601] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5636.543245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5636.941596] Lustre: lustre-OST0000: new disk, initializing [ 5636.946247] Lustre: Skipped 1 previous similar message [ 5636.957544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5636.960322] Lustre: Skipped 2 previous similar messages [ 5636.967922] Lustre: 175050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5644.775672] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5645.993444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5646.018304] Lustre: Skipped 1 previous similar message [ 5646.027598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5646.217529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5655.887309] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5670.543237] Lustre: server umount lustre-OST0000 complete [ 5677.031644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5677.035509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5677.049514] Lustre: Skipped 2 previous similar messages [ 5677.076980] Lustre: Skipped 1 previous similar message [ 5688.295031] Lustre: 155659:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753266/real 1786753266] req@000000002e3fd9c3 x1873545738853888/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786753282 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5688.335825] Lustre: 155659:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5690.337455] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5690.636470] Lustre: server umount lustre-MDT0000 complete [ 5693.408828] Lustre: 155661:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753271/real 1786753271] req@000000004ba6d7a8 x1873545738855552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786753287 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5693.410106] LustreError: 173075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.470864] LustreError: 173075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5695.051084] LustreError: 173061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753289 with bad export cookie 10693166185022246994 [ 5695.052476] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5695.063455] LustreError: 173061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5695.461846] Lustre: server umount lustre-MDT0001 complete [ 5705.687191] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:21:38 (1786753298) [ 5723.836274] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 5734.319180] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5776.560465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5776.817555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5776.842783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5776.927577] Lustre: lustre-MDT0000: new disk, initializing [ 5777.017859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5777.027518] Lustre: Skipped 2 previous similar messages [ 5777.049877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5782.810133] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5794.670167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5794.820341] Lustre: 179378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5795.158515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5800.135203] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5810.956920] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5817.079681] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5825.034092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5825.523540] Lustre: 181355:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5832.847156] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5833.854083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5834.038913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5844.038656] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5854.699512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5854.700386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5854.720824] Lustre: Skipped 1 previous similar message [ 5854.725155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5854.736906] Lustre: Skipped 7 previous similar messages [ 5856.859896] Lustre: server umount lustre-OST0000 complete [ 5861.866242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5861.879624] LustreError: Skipped 1 previous similar message [ 5867.446843] Lustre: server umount lustre-MDT0000 complete [ 5870.053366] LustreError: 179383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.081836] LustreError: 179383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5871.987592] LustreError: 179370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753466 with bad export cookie 10693166185022248114 [ 5871.994608] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5872.002370] LustreError: 179370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5872.742315] Lustre: server umount lustre-MDT0001 complete [ 5888.050318] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 5892.097813] Key type lgssc unregistered [ 5892.357074] LNet: 183218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5892.366767] LNetError: 183218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5892.412817] LNet: Removed LNI 192.168.202.118@tcp [ 5893.402311] Key type .llcrypt unregistered [ 5893.406365] Key type ._llcrypt unregistered [ 5924.419379] Key type ._llcrypt registered [ 5924.421536] Key type .llcrypt registered [ 5924.555548] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5926.076372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5926.109027] alg: No test for adler32 (adler32-zlib) [ 5927.242859] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 5927.513837] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 5929.160474] Key type lgssc registered [ 5930.335640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5938.839668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5938.877805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.660909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5945.535515] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5950.959862] LustreError: 184638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5950.991676] LustreError: 184638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5954.786028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5955.397772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5961.675661] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5971.298700] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5979.018829] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5987.620731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5988.158838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5996.028905] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6003.956213] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6010.685250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6031.339283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6031.355680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6031.384893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6031.394932] Lustre: Skipped 1 previous similar message [ 6036.214647] Lustre: server umount lustre-OST0000 complete [ 6041.581454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6041.593411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6041.597981] Lustre: Skipped 2 previous similar messages [ 6041.618182] Lustre: Skipped 1 previous similar message [ 6046.690774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6046.696406] Lustre: Skipped 1 previous similar message [ 6047.110307] Lustre: server umount lustre-MDT0000 complete [ 6050.907316] LustreError: 184618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753645 with bad export cookie 17233433252220773972 [ 6050.916240] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6051.126547] Lustre: server umount lustre-MDT0001 complete [ 6061.965234] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 6064.539717] Key type lgssc unregistered [ 6064.824775] LNet: 188204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6064.830583] LNetError: 188204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6065.895407] LNet: Removed LNI 192.168.202.118@tcp [ 6067.008222] Key type .llcrypt unregistered [ 6067.011037] Key type ._llcrypt unregistered [ 6092.924535] Key type ._llcrypt registered [ 6092.926051] Key type .llcrypt registered [ 6093.080113] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 6094.485972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6094.555306] alg: No test for adler32 (adler32-zlib) [ 6096.415476] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 6096.897958] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 6098.609230] Key type lgssc registered [ 6100.620029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6112.742180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6112.762449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6114.534549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6120.199445] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6125.029775] LustreError: 189621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6125.071100] LustreError: 189621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6130.149434] LustreError: 189622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6130.775395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6131.166752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6136.542724] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6145.471818] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6152.288705] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6160.257689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6160.741088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6167.571280] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6175.914224] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6185.360778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6192.813087] Lustre: lustre-OST0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 6195.688617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6195.695266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6195.696779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6195.702509] Lustre: Skipped 1 previous similar message [ 6195.723673] Lustre: Skipped 1 previous similar message [ 6197.895806] Lustre: lustre-OST0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 6198.685241] Lustre: server umount lustre-OST0000 complete [ 6206.546586] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff995f90c4d800.ost_server_uuid 50 [ 6208.601394] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6222.708152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6222.730221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6227.940761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6227.947843] Lustre: Skipped 2 previous similar messages [ 6229.120988] Lustre: server umount lustre-MDT0000 complete [ 6232.596567] LustreError: 189603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753827 with bad export cookie 15874084330073612329 [ 6232.598326] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6232.612443] LustreError: 189603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6233.044819] Lustre: server umount lustre-MDT0001 complete [ 6242.466234] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:30:35 (1786753835) [ 6250.111511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6250.757866] LustreError: 193563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6250.880037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6255.642341] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6256.097898] LustreError: 193564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6261.219206] LustreError: 193563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.405287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.748938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6269.955542] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6279.499456] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6287.416213] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6295.321440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6295.527630] LustreError: 195387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6295.780677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6301.204482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6302.598403] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6311.850864] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6321.014499] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6321.351177] Lustre: lustre-OST0001: new disk, initializing [ 6321.359558] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6321.376134] Lustre: 196407:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6321.514125] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6329.096789] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6331.552479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6331.575695] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6331.703062] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6340.398309] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6346.495402] Lustre: lustre-OST0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 6347.233818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6347.250272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6347.273318] Lustre: Skipped 1 previous similar message [ 6351.142630] Lustre: server umount lustre-OST0000 complete [ 6351.514912] LustreError: 195388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6358.645985] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6358.853252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6364.052600] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6364.376883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6383.078740] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6383.091745] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6383.115974] Lustre: Skipped 1 previous similar message [ 6383.143565] LustreError: Skipped 1 previous similar message [ 6383.145460] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6383.156485] Lustre: Skipped 3 previous similar messages [ 6385.864789] Lustre: server umount lustre-OST0001 complete [ 6393.317452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.347478] Lustre: Skipped 2 previous similar messages [ 6396.411398] Lustre: server umount lustre-MDT0000 complete [ 6398.441447] LustreError: 194580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6398.474758] LustreError: 194580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 6400.838708] LustreError: 193543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753995 with bad export cookie 15874084330073613400 [ 6400.841482] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6400.851656] LustreError: 193543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6401.283110] Lustre: server umount lustre-MDT0001 complete [ 6429.285257] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:33:42 (1786754022) [ 6437.150738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6437.513766] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6437.545781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6437.684372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6443.283741] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6453.047620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6453.163132] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6453.235271] Lustre: 199849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6458.671918] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6467.027909] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6473.952555] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6480.842444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6481.101110] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6487.702352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6488.087182] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6496.366174] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6502.566125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6502.621393] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6502.806396] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6502.810668] Lustre: Skipped 2 previous similar messages [ 6510.239884] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6518.187740] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6523.362822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6523.372216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6523.383620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6523.388855] Lustre: Skipped 2 previous similar messages [ 6528.731952] Lustre: server umount lustre-OST0000 complete [ 6529.686098] LustreError: 201703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.920028] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6546.925187] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6546.943040] Lustre: Skipped 2 previous similar messages [ 6546.970104] LustreError: Skipped 2 previous similar messages [ 6551.048924] Lustre: server umount lustre-OST0001 complete [ 6563.999719] Lustre: server umount lustre-MDT0000 complete [ 6568.624096] LustreError: 199842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754163 with bad export cookie 15874084330073614807 [ 6568.651512] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6568.657568] LustreError: 199842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6569.069945] Lustre: server umount lustre-MDT0001 complete [ 6594.986771] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:36:28 (1786754188) [ 6612.815489] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 6622.498241] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 6659.310138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.510508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6659.538661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6659.646138] Lustre: lustre-MDT0000: new disk, initializing [ 6659.711548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6659.723908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6664.315288] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6675.044590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6675.215416] Lustre: 207948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6675.269410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6675.272407] Lustre: Skipped 1 previous similar message [ 6675.372873] Lustre: lustre-MDT0001: new disk, initializing [ 6675.618095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6675.627730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6680.075563] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6688.892088] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6694.726399] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6703.262761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6703.718262] Lustre: lustre-OST0000: new disk, initializing [ 6703.725482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6703.736627] Lustre: 209920:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6711.388256] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6711.912961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6711.933286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6712.074505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6722.652557] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6728.446384] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6728.789850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6734.406856] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6734.706854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6737.893934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6737.901158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6737.920428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6737.920438] Lustre: Skipped 10 previous similar messages [ 6737.960432] Lustre: Skipped 3 previous similar messages [ 6742.990249] Lustre: server umount lustre-OST0000 complete [ 6753.836113] Lustre: server umount lustre-MDT0000 complete [ 6757.552473] LustreError: 207940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754352 with bad export cookie 15874084330073616368 [ 6757.558413] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6757.562138] LustreError: 207940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6757.985170] Lustre: server umount lustre-MDT0001 complete [ 6767.956802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6768.369514] LustreError: 212139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6768.404742] LustreError: 212139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 6772.516653] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6782.516815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6787.089474] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6795.259953] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6800.812832] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6808.337054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6816.090912] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6823.775650] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6829.136948] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6829.418907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6835.528626] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6835.778891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6838.756790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6838.782049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6838.809033] Lustre: Skipped 2 previous similar messages [ 6844.179992] Lustre: server umount lustre-OST0000 complete [ 6851.548388] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6856.379767] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6888.567513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6893.993427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6894.013641] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6894.014917] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6894.048090] Lustre: Skipped 1 previous similar message [ 6896.755633] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6905.086132] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6906.724324] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6914.533825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6914.538396] LustreError: Skipped 2 previous similar messages [ 6914.539176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6914.547833] Lustre: Skipped 8 previous similar messages [ 6916.634617] Lustre: server umount lustre-OST0000 complete [ 6928.923026] Lustre: server umount lustre-MDT0000 complete [ 6933.347697] LustreError: 212119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754527 with bad export cookie 15874084330073617292 [ 6933.358045] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.361105] LustreError: 212119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6933.996578] Lustre: server umount lustre-MDT0001 complete [ 6943.880909] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:42:16 (1786754536) [ 6951.177413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6951.805344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6951.817787] Lustre: Skipped 6 previous similar messages [ 6957.143338] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6966.617597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6972.704816] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6980.529951] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6988.648771] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7000.370441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7009.088210] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7018.315511] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7024.399862] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7024.768337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7029.725402] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7029.932300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7036.687252] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7036.910506] Lustre: lustre-OST0001: new disk, initializing [ 7036.913930] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7036.918331] Lustre: 221485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7044.812084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7044.832388] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7044.955182] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7050.245919] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7057.343160] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7063.529188] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7063.837333] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7068.968884] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7069.380200] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7075.813265] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7075.819615] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7075.833393] Lustre: Skipped 5 previous similar messages [ 7075.838313] LustreError: Skipped 1 previous similar message [ 7077.716513] Lustre: server umount lustre-OST0001 complete [ 7080.944131] LustreError: 219937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7080.963029] LustreError: 219937:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 7085.917497] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7086.203287] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7091.708946] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7091.985312] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7122.636186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7128.056313] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7128.073322] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7128.078884] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7129.589280] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7138.367285] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7139.670419] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7147.759564] Lustre: server umount lustre-OST0001 complete [ 7174.126613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7174.134823] Lustre: Skipped 10 previous similar messages [ 7180.256269] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7180.475580] Lustre: server umount lustre-OST0000 complete [ 7191.323674] Lustre: server umount lustre-MDT0000 complete [ 7196.165290] LustreError: 218091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754790 with bad export cookie 15874084330073618258 [ 7196.168809] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7196.174528] LustreError: 218091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7202.671429] Lustre: server umount lustre-MDT0001 complete [ 7229.786091] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:47:02 (1786754822) [ 7237.585278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7237.890902] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7237.955932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7243.667482] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7253.843515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7253.917622] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7253.936592] Lustre: 226737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7259.336497] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7268.477506] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7275.355395] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7282.575577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7282.733333] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7291.901089] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7301.072331] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7312.854725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7312.905319] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7321.043935] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7329.775451] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7331.973973] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7338.847306] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7339.258934] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7344.791208] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7345.068616] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7353.706329] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff995f892a8800.ost_server_uuid 50 [ 7355.856367] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff995f892a8800.ost_server_uuid in IDLE state after 0 sec [ 7357.270509] Lustre: Permanently deactivating lustre-OST0001 [ 7357.286577] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7361.317745] Lustre: Permanently reactivating lustre-OST0001 [ 7361.336632] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7361.340892] Lustre: Skipped 2 previous similar messages [ 7367.153958] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7367.172637] LustreError: Skipped 3 previous similar messages [ 7367.181581] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7367.201164] Lustre: Skipped 6 previous similar messages [ 7372.823529] Lustre: server umount lustre-OST0001 complete [ 7383.779052] Lustre: server umount lustre-OST0000 complete [ 7394.300874] Lustre: server umount lustre-MDT0000 complete [ 7398.031318] LustreError: 226729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754992 with bad export cookie 15874084330073619546 [ 7398.041849] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7398.043285] LustreError: 226729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7398.534927] Lustre: server umount lustre-MDT0001 complete [ 7426.446335] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:50:19 (1786755019) [ 7435.773255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7436.113435] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7436.147482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7436.156178] Lustre: Skipped 2 previous similar messages [ 7441.204593] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7450.607575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.732093] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7450.759392] Lustre: 233438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7457.615719] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7467.105311] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7474.150182] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7481.440141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7481.693701] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7481.729529] Lustre: Permanently deactivating lustre-OST0000 [ 7481.745085] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7482.051276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.062016] Lustre: Skipped 10 previous similar messages [ 7489.713595] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7491.614287] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7498.081253] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7506.230797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7506.302671] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7514.852572] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7515.329104] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7523.200680] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7525.306457] Lustre: Permanently reactivating lustre-OST0000 [ 7525.328849] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7525.336089] Lustre: Skipped 2 previous similar messages [ 7534.116503] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7534.122944] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 00000000aefcff3c, cur 1786755128 deadline 1786755116 last 1786755086 [ 7534.126823] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7534.135311] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7534.170090] Lustre: Skipped 1 previous similar message [ 7534.224423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7542.996470] Lustre: server umount lustre-OST0001 complete [ 7553.684518] Lustre: server umount lustre-OST0000 complete [ 7564.358250] Lustre: server umount lustre-MDT0000 complete [ 7569.404238] LustreError: 234181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755163 with bad export cookie 15874084330073621212 [ 7569.407798] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7569.420548] LustreError: 234181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7569.841741] Lustre: server umount lustre-MDT0001 complete [ 7581.277435] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 7584.194495] Key type lgssc unregistered [ 7584.453860] LNet: 238306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7584.464743] LNetError: 238306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7584.481627] LNet: Removed LNI 192.168.202.118@tcp [ 7585.561616] Key type .llcrypt unregistered [ 7585.563464] Key type ._llcrypt unregistered [ 7607.957603] Key type ._llcrypt registered [ 7607.959366] Key type .llcrypt registered [ 7608.078618] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7609.911648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7609.967763] alg: No test for adler32 (adler32-zlib) [ 7611.107776] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 7611.311119] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 7612.956359] Key type lgssc registered [ 7614.361910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7626.401854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7626.425090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.137390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7633.228202] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7637.475145] LustreError: 239771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7637.503873] LustreError: 239771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7642.618356] LustreError: 239772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7645.034099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.180310] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7645.205764] Lustre: 239760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7645.215431] Lustre: Permanently deactivating lustre-MDT0001 [ 7645.229430] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7645.529237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7651.642705] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7651.904351] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7660.947633] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7670.369343] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7679.733556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7680.231506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7688.439471] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7690.480788] LustreError: 241615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7695.852974] LustreError: 241617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7697.902765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7698.199559] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7703.013028] LustreError: 241616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7703.071910] LustreError: 241616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7706.592524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7707.082386] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7712.272405] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7712.282784] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7715.511607] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7724.393440] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7728.891561] Lustre: Permanently reactivating lustre-MDT0001 [ 7728.919465] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7728.928522] Lustre: Skipped 2 previous similar messages [ 7731.048425] Lustre: lustre-MDT0001: Client 57114238-a9d3-4d6f-8695-4667a79e6592 (at 192.168.202.18@tcp) reconnecting [ 7737.908817] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7739.596986] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7739.609035] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7739.616627] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7739.637464] LustreError: 243562:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7739.643307] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7739.652444] LustreError: 243562:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7739.652601] LustreError: 243562:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.118@tcp: cfg command failed: rc = -108 [ 7739.656819] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7739.656819] [ 7739.660189] LustreError: 239763:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7739.764317] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7747.421890] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7747.689989] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7754.110761] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7754.534078] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7757.972373] Lustre: Permanently deactivating lustre-MDT0001 [ 7758.050340] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7758.061885] Lustre: Skipped 2 previous similar messages [ 7766.794391] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7773.666918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7773.688094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7773.731584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7775.013028] Lustre: server umount lustre-MDT0000 complete [ 7775.716136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7775.718056] LustreError: 239772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7775.735429] Lustre: Skipped 2 previous similar messages [ 7775.779287] LustreError: 239772:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7779.907297] LustreError: 239752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755374 with bad export cookie 11833785975669318129 [ 7779.913452] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7780.245403] Lustre: server umount lustre-MDT0001 complete [ 7787.555548] Lustre: server umount lustre-OST0000 complete [ 7803.039784] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:56:35 (1786755395) [ 7814.185480] Lustre: server umount lustre-OST0001 complete [ 7820.940360] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 7831.211417] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7867.199189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7867.419899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7867.428191] Lustre: Skipped 2 previous similar messages [ 7867.445408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7867.518631] Lustre: lustre-MDT0000: new disk, initializing [ 7867.589731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7867.610485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7872.126416] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7882.816767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.990705] Lustre: 248038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7883.042993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7883.047525] Lustre: Skipped 1 previous similar message [ 7883.126793] Lustre: lustre-MDT0001: new disk, initializing [ 7883.216878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7883.257995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7883.266658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7887.836905] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7898.181485] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7904.329232] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7911.723370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7912.179826] Lustre: lustre-OST0000: new disk, initializing [ 7912.188279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7912.199610] Lustre: 250009:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7912.349482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7917.694548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7917.712778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7917.874425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7918.384411] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7929.159972] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7931.362472] LustreError: 248046:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7941.008356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7941.191385] Lustre: lustre-OST0001: new disk, initializing [ 7941.194954] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7941.200441] Lustre: 251122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7941.373356] LustreError: 248046:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7941.455221] LustreError: 249751:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7948.947772] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7949.967293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7949.980652] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7950.145574] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7951.544151] LustreError: 249751:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7960.210994] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7962.177285] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7965.665607] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7965.668348] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7965.678743] LustreError: Skipped 1 previous similar message [ 7965.681021] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7965.708789] Lustre: Skipped 1 previous similar message [ 7965.742230] Lustre: Skipped 1 previous similar message [ 7970.798210] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7970.807528] Lustre: Skipped 1 previous similar message [ 7975.920440] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7975.927060] Lustre: Skipped 1 previous similar message [ 7978.976196] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7979.078073] Lustre: server umount lustre-OST0001 complete [ 7981.039939] LustreError: 249995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7981.052032] LustreError: 249995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7999.979345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7999.979638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7999.985562] LustreError: Skipped 1 previous similar message [ 8000.001670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8002.795774] Lustre: server umount lustre-OST0000 complete [ 8011.234089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8011.246731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8011.270472] Lustre: Skipped 1 previous similar message [ 8011.285961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8011.288711] Lustre: Skipped 3 previous similar messages [ 8013.186484] Lustre: server umount lustre-MDT0000 complete [ 8016.702119] LustreError: 248031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755611 with bad export cookie 11833785975669320726 [ 8016.714220] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8016.714978] LustreError: 248031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8031.200626] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8031.472845] Lustre: server umount lustre-MDT0001 complete [ 8040.727136] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8042.971580] Key type lgssc unregistered [ 8043.269877] LNet: 253159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8043.285044] LNetError: 253159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8043.313532] LNet: Removed LNI 192.168.202.118@tcp [ 8044.170548] Key type .llcrypt unregistered [ 8044.173553] Key type ._llcrypt unregistered [ 8086.775662] Key type ._llcrypt registered [ 8086.782683] Key type .llcrypt registered [ 8086.945517] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8088.403954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8088.444344] alg: No test for adler32 (adler32-zlib) [ 8089.439848] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 8089.715767] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8091.560199] Key type lgssc registered [ 8093.169505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8102.773290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8102.805083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8104.027278] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8104.084692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8104.194683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8109.088872] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8117.119201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8117.216609] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8117.247567] Lustre: 255380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8117.536111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8121.710347] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8129.055720] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8134.913388] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8141.031670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8141.186179] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8141.371897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8147.382160] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8150.686108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8154.665848] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8206.818088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8206.828819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8206.848846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8207.329484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8207.373454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8207.383906] Lustre: Skipped 1 previous similar message [ 8212.123738] Lustre: server umount lustre-OST0000 complete [ 8216.212341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8228.762367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8229.046211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8233.147416] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8233.174426] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8235.329218] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8241.994098] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8247.547968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8247.574974] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8283.621704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8283.623230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8283.627662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8283.643120] Lustre: Skipped 1 previous similar message [ 8283.653361] Lustre: Skipped 1 previous similar message [ 8287.125330] Lustre: server umount lustre-OST0000 complete [ 8291.809990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8291.827144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8291.844713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8297.694679] Lustre: server umount lustre-MDT0000 complete [ 8300.963063] LustreError: 255374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755895 with bad export cookie 580664538313208998 [ 8300.968186] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8300.973317] LustreError: 255374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8301.389541] Lustre: server umount lustre-MDT0001 complete [ 8310.772651] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8313.268738] Key type lgssc unregistered [ 8313.556154] LNet: 260810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8313.566527] LNetError: 260810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8313.589176] LNet: Removed LNI 192.168.202.118@tcp [ 8314.438278] Key type .llcrypt unregistered [ 8314.440813] Key type ._llcrypt unregistered [ 8339.312404] Key type ._llcrypt registered [ 8339.314630] Key type .llcrypt registered [ 8339.395890] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8340.289674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8340.414635] alg: No test for adler32 (adler32-zlib) [ 8341.642916] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 8341.834982] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8343.472490] Key type lgssc registered [ 8345.332052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8355.304917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8355.329786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8357.064580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8361.925727] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8366.632169] LustreError: 262226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8366.646416] LustreError: 262226:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8371.681517] LustreError: 262227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8371.722253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8371.993463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8376.337267] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8384.126872] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8390.700548] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8398.873509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8399.353307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8402.429591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8405.635351] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8412.515986] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8442.337776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8442.343471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8442.359073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8442.377175] Lustre: Skipped 1 previous similar message [ 8445.861327] Lustre: server umount lustre-OST0000 complete [ 8452.579985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8452.584522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8452.605741] Lustre: Skipped 1 previous similar message [ 8452.632225] Lustre: Skipped 2 previous similar messages [ 8457.699589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8457.704271] Lustre: Skipped 1 previous similar message [ 8462.819914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8464.352212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8464.598164] Lustre: server umount lustre-MDT0000 complete [ 8467.942028] LustreError: 262227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8467.970852] LustreError: 262227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8468.790853] LustreError: 262207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756063 with bad export cookie 4263539967348022329 [ 8468.793024] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8468.799909] LustreError: 262207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8469.133726] Lustre: server umount lustre-MDT0001 complete [ 8482.020357] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8484.724723] Key type lgssc unregistered [ 8485.090190] LNet: 266540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8485.101523] LNetError: 266540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8485.121791] LNet: Removed LNI 192.168.202.118@tcp [ 8486.273635] Key type .llcrypt unregistered [ 8486.275275] Key type ._llcrypt unregistered [ 8505.199858] Key type ._llcrypt registered [ 8505.201656] Key type .llcrypt registered [ 8505.361182] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8506.582634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8506.811152] alg: No test for adler32 (adler32-zlib) [ 8507.999544] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 8508.266481] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8509.984231] Key type lgssc registered [ 8511.154596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8518.449936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8518.483729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8520.072021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8524.360980] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8530.411249] LustreError: 267762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8530.422660] LustreError: 267762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8532.604978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8532.950928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8537.597166] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8546.396630] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8552.584760] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8559.161465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8559.468452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8564.596312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8564.902821] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8572.567971] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8588.267133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8588.273165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8588.301530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8588.328922] Lustre: Skipped 1 previous similar message [ 8593.387135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8594.436797] Lustre: server umount lustre-OST0000 complete [ 8599.521843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8599.534602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8599.546277] Lustre: Skipped 1 previous similar message [ 8599.560493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8599.563180] Lustre: Skipped 1 previous similar message [ 8603.619887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8603.620316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8603.657811] Lustre: Skipped 1 previous similar message [ 8608.738844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8608.751551] Lustre: Skipped 1 previous similar message [ 8613.344722] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8613.585965] Lustre: server umount lustre-MDT0000 complete [ 8613.860912] LustreError: 267763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8613.883236] LustreError: 267763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8617.897879] LustreError: 267743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756212 with bad export cookie 8419750775627576678 [ 8617.906926] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8618.201660] Lustre: server umount lustre-MDT0001 complete [ 8628.388783] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8631.028712] Key type lgssc unregistered [ 8631.323390] LNet: 271448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8631.330602] LNetError: 271448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8631.349309] LNet: Removed LNI 192.168.202.118@tcp [ 8632.362917] Key type .llcrypt unregistered [ 8632.365800] Key type ._llcrypt unregistered [ 8657.539797] Key type ._llcrypt registered [ 8657.541612] Key type .llcrypt registered [ 8657.696326] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8659.371195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8659.492079] alg: No test for adler32 (adler32-zlib) [ 8660.661132] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 8660.991953] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8662.633726] Key type lgssc registered [ 8663.983772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8673.095195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8673.135049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8674.690580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8679.895165] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8685.027957] LustreError: 272864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8685.043919] LustreError: 272864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8687.915884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.307644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8693.284816] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8701.533854] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8707.239428] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8714.945317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8715.315472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8720.443623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8721.217716] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8729.469621] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8758.250037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8758.253378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8758.285030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8758.309500] Lustre: Skipped 1 previous similar message [ 8763.366723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8763.374026] Lustre: Skipped 1 previous similar message [ 8764.336212] Lustre: server umount lustre-OST0000 complete [ 8770.530572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8770.545467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8770.570259] Lustre: Skipped 1 previous similar message [ 8770.582286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8773.602182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8773.615557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8778.732575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8778.739590] Lustre: Skipped 2 previous similar messages [ 8783.328753] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8783.614986] Lustre: server umount lustre-MDT0000 complete [ 8783.848069] LustreError: 272863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8787.746946] LustreError: 272842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756382 with bad export cookie 6624231418297186841 [ 8787.762370] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8787.766137] LustreError: 272842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8788.157443] Lustre: server umount lustre-MDT0001 complete [ 8800.989476] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8803.895815] Key type lgssc unregistered [ 8804.195949] LNet: 277175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8804.202543] LNetError: 277175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8804.243574] LNet: Removed LNI 192.168.202.118@tcp [ 8805.234830] Key type .llcrypt unregistered [ 8805.239960] Key type ._llcrypt unregistered [ 8825.050664] Key type ._llcrypt registered [ 8825.054752] Key type .llcrypt registered [ 8825.185694] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8826.670086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8826.734289] alg: No test for adler32 (adler32-zlib) [ 8827.847651] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 8828.095283] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8829.816209] Key type lgssc registered [ 8831.113217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8840.046991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8840.069911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8841.843060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8847.783592] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8852.461593] LustreError: 278417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8852.480170] LustreError: 278417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8857.582608] LustreError: 278419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8857.869682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8858.158419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8863.716421] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8872.154873] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8878.776251] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8888.190770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8888.615218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8895.347628] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8903.242584] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8913.212528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8928.737783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8928.752985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.769971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8928.785476] Lustre: Skipped 1 previous similar message [ 8933.229526] Lustre: server umount lustre-OST0000 complete [ 8938.978367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8939.001327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8939.013464] Lustre: Skipped 2 previous similar messages [ 8943.931803] Lustre: server umount lustre-MDT0000 complete [ 8944.109612] LustreError: 278421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8944.126982] LustreError: 278421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8947.860531] LustreError: 278382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756542 with bad export cookie 5562748518618937740 [ 8947.873978] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8947.874604] LustreError: 278382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8948.219205] Lustre: server umount lustre-MDT0001 complete [ 8960.296900] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8963.187085] Key type lgssc unregistered [ 8963.443181] LNet: 282145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8963.453368] LNetError: 282145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8963.469699] LNet: Removed LNI 192.168.202.118@tcp [ 8964.301645] Key type .llcrypt unregistered [ 8964.303983] Key type ._llcrypt unregistered [ 8977.428749] Key type ._llcrypt registered [ 8977.435657] Key type .llcrypt registered [ 8977.543201] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8995.862706] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 9010.695511] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9011.451501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9011.509087] alg: No test for adler32 (adler32-zlib) [ 9012.508646] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 9012.709054] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 9014.435757] Key type lgssc registered [ 9015.708431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9054.195797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9054.211151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9055.459199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9055.504697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9055.592317] Lustre: lustre-MDT0000: new disk, initializing [ 9055.758469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9055.801518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9060.241506] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9071.640489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9071.796759] Lustre: 285695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9071.869592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9071.872269] Lustre: Skipped 1 previous similar message [ 9072.025249] Lustre: lustre-MDT0001: new disk, initializing [ 9072.237495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9072.319913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9072.331847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9077.534884] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9086.889556] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9093.016901] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9099.785774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9100.151665] Lustre: lustre-OST0000: new disk, initializing [ 9100.156971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9100.166202] Lustre: 287667:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9100.274057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9106.523136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9106.531139] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9106.536918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9106.639215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9115.697389] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9120.646439] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9120.884628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9124.901160] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9125.088555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9127.394024] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9127.402627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9127.409238] Lustre: Skipped 1 previous similar message [ 9127.409853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9127.415911] Lustre: Skipped 1 previous similar message [ 9132.513707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9132.522731] Lustre: Skipped 1 previous similar message [ 9137.652576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9137.662323] Lustre: Skipped 1 previous similar message [ 9141.216263] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9141.346987] Lustre: server umount lustre-OST0000 complete [ 9147.874691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9147.877851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9147.890193] Lustre: Skipped 1 previous similar message [ 9151.101416] Lustre: server umount lustre-MDT0000 complete [ 9154.742084] LustreError: 285688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756749 with bad export cookie 3883179248964221556 [ 9154.746614] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9154.749654] LustreError: 285688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9155.000495] Lustre: server umount lustre-MDT0001 complete [ 9165.710201] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:19:18 (1786756758) [ 9174.661890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9175.418558] LustreError: 290081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9175.455989] LustreError: 290081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9175.582304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9180.658741] LustreError: 290082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9180.853695] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9185.768342] LustreError: 290081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9190.885266] LustreError: 290082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9191.240358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9191.824981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9196.620391] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9204.797086] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9210.595621] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9217.128917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9217.450673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9224.054705] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9232.171403] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9244.131564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9244.158695] LustreError: Skipped 1 previous similar message [ 9244.182335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9244.254118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9244.279292] Lustre: Skipped 1 previous similar message [ 9246.177557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9246.203662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9250.093371] Lustre: server umount lustre-OST0000 complete [ 9256.417683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9256.437053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9256.453322] Lustre: Skipped 3 previous similar messages [ 9262.055590] Lustre: server umount lustre-MDT0000 complete [ 9266.398834] LustreError: 290062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756860 with bad export cookie 3883179248964222473 [ 9266.417655] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9267.077511] Lustre: server umount lustre-MDT0001 complete [ 9281.005744] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 9284.529164] Key type lgssc unregistered [ 9284.864255] LNet: 293623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9284.875258] LNetError: 293623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9284.898626] LNet: Removed LNI 192.168.202.118@tcp [ 9285.984611] Key type .llcrypt unregistered [ 9285.987524] Key type ._llcrypt unregistered [ 9322.279906] Key type ._llcrypt registered [ 9322.282308] Key type .llcrypt registered [ 9322.385197] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 9340.139985] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9341.432727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9341.492053] alg: No test for adler32 (adler32-zlib) [ 9342.694137] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 9343.016933] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 9344.664165] Key type lgssc registered [ 9345.771067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9390.113321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9390.169634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9391.432845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9391.471490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9391.557438] Lustre: lustre-MDT0000: new disk, initializing [ 9391.634322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9391.648436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9395.334471] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9405.800668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9405.943304] Lustre: 297446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9406.002504] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9406.007463] Lustre: Skipped 1 previous similar message [ 9406.058351] Lustre: lustre-MDT0001: new disk, initializing [ 9406.125087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9406.147283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9406.155544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9410.421315] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9420.619728] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9426.024478] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9432.795264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9432.996780] Lustre: lustre-OST2710: new disk, initializing [ 9432.999859] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9433.004527] Lustre: 299418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9433.090980] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9439.733798] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9440.824558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9440.830956] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9440.955774] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9450.154461] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9458.048513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9458.295474] Lustre: lustre-OST03e8: new disk, initializing [ 9458.300797] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9458.309789] Lustre: 300482:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9458.456232] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9465.794746] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9467.504685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9467.525764] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9467.689666] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9474.521859] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9482.870401] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9483.062760] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9488.742373] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9488.914741] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9494.620310] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9494.862782] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9500.548625] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9500.741212] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9507.616754] Lustre: server umount lustre-MDT0000 complete [ 9508.840885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9508.861395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9512.042054] LustreError: 297440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757106 with bad export cookie 14317316198594813457 [ 9512.052700] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9512.065540] LustreError: 297440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9512.653954] Lustre: server umount lustre-MDT0001 complete [ 9527.595757] Lustre: server umount lustre-OST2710 complete [ 9529.826193] Lustre: 295250:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786757108/real 1786757108] req@00000000c7e9917c x1873550442045184/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786757124 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9529.859410] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9529.869594] Lustre: Skipped 3 previous similar messages [ 9533.506983] Lustre: server umount lustre-OST03e8 complete [ 9557.433858] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 9567.977414] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9614.380083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9614.541249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9614.565619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9614.618251] Lustre: lustre-MDT0000: new disk, initializing [ 9614.671320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9614.690623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9619.255418] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9630.238668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9630.394622] Lustre: 305736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9630.440918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9630.444237] Lustre: Skipped 1 previous similar message [ 9630.500275] Lustre: lustre-MDT0001: new disk, initializing [ 9630.565588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9630.601979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9630.654777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9636.001605] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9645.825837] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9652.161297] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9659.785729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9660.013760] Lustre: lustre-OST0000: new disk, initializing [ 9660.015734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9660.018905] Lustre: 307705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9660.068787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9665.592451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9665.602713] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9665.673707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9665.817551] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9677.013881] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9681.596169] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9681.800189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9686.535992] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9686.962288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9691.114126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9691.126556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9691.131262] Lustre: Skipped 1 previous similar message [ 9696.232073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9696.239443] Lustre: Skipped 1 previous similar message [ 9701.357196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9701.378578] Lustre: Skipped 1 previous similar message [ 9703.393040] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9703.715389] Lustre: server umount lustre-OST0000 complete [ 9712.613557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9712.624892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9712.657218] Lustre: Skipped 1 previous similar message [ 9712.668689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9716.705665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9716.707666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9719.353694] Lustre: server umount lustre-MDT0000 complete [ 9721.827088] LustreError: 305746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9721.840759] LustreError: 305746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9723.999065] LustreError: 306530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757318 with bad export cookie 14317316198594815179 [ 9724.005702] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9724.014601] LustreError: 306530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9724.510664] Lustre: server umount lustre-MDT0001 complete [ 9735.108318] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:28:47 (1786757327) [ 9736.980426] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9739.438890] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:28:52 (1786757332) [ 9743.075056] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9773.747243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9774.078726] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9774.098564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9774.222165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9778.963849] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9788.741159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9788.906373] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9788.947112] Lustre: 311565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9793.526974] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9817.271476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9817.665206] Lustre: Denying initial registration attempt from nid 192.168.202.118@tcp, specified as failover [ 9817.678240] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9817.685570] LustreError: 313022:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9817.698482] LustreError: 313022:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9817.709228] LustreError: 313022:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9817.826789] Lustre: server umount lustre-OST0000 complete [ 9817.836810] LustreError: 313022:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9826.788441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9826.819436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9826.837872] Lustre: Skipped 2 previous similar messages [ 9832.292175] Lustre: server umount lustre-MDT0000 complete [ 9837.026362] LustreError: 311556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757431 with bad export cookie 14317316198594816103 [ 9837.036954] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9837.043900] LustreError: 311556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9837.527606] Lustre: server umount lustre-MDT0001 complete [ 9851.177453] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 9854.144994] Key type lgssc unregistered [ 9854.385129] LNet: 314145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9854.395038] LNetError: 314145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9854.419389] LNet: Removed LNI 192.168.202.118@tcp [ 9855.299444] Key type .llcrypt unregistered [ 9855.300938] Key type ._llcrypt unregistered [ 9868.934477] Key type ._llcrypt registered [ 9868.940983] Key type .llcrypt registered [ 9869.025892] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9870.155991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9870.255425] alg: No test for adler32 (adler32-zlib) [ 9871.348765] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 9871.669995] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 9873.440188] Key type lgssc registered [ 9874.959548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9930.151982] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9939.912636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9939.943628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9941.216853] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9941.233189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9941.338386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9946.273384] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9955.062463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9955.204315] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9955.265559] Lustre: 316931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9955.693487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9961.312762] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9969.364868] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9977.621479] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9999.286786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9999.502754] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9999.755452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10007.638667] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10017.373777] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10022.882163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10022.896250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10022.901259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10022.914390] Lustre: Skipped 1 previous similar message [10022.928238] Lustre: Skipped 1 previous similar message [10028.009087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10028.020123] Lustre: Skipped 1 previous similar message [10033.125880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10033.138125] Lustre: Skipped 1 previous similar message [10034.145241] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10034.368854] Lustre: server umount lustre-OST0000 complete [10042.851308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10042.875633] LustreError: Skipped 1 previous similar message [10042.877073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10042.900471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10048.481806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10048.494017] Lustre: Skipped 3 previous similar messages [10053.088349] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10053.264259] Lustre: server umount lustre-MDT0000 complete [10056.964695] LustreError: 316923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757651 with bad export cookie 7849841545395474698 [10056.973478] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10056.979046] LustreError: 316923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10057.015082] LustreError: 314823:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [10057.481147] Lustre: server umount lustre-MDT0001 complete [10068.681729] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [10072.252777] Key type lgssc unregistered [10072.601409] LNet: 320511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10072.609580] LNetError: 320511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10073.638990] LNet: Removed LNI 192.168.202.118@tcp [10074.695104] Key type .llcrypt unregistered [10074.706813] Key type ._llcrypt unregistered [10106.796241] Key type ._llcrypt registered [10106.798881] Key type .llcrypt registered [10106.939616] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [10108.129800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10108.360023] alg: No test for adler32 (adler32-zlib) [10109.466187] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [10109.695404] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [10111.352158] Key type lgssc registered [10112.892185] Lustre: Echo OBD driver; http://www.lustre.org/ [10122.335266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10122.347130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10123.808873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10128.909063] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10133.988125] LustreError: 321931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10134.025076] LustreError: 321931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10139.113539] LustreError: 321930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10140.483483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10141.491179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10147.694323] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10157.545890] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10164.849991] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10174.068399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10174.541531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10182.036685] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10191.473319] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10207.949323] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [10208.210708] Lustre: server umount lustre-MDT0000 complete [10208.236314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10208.241682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10208.247700] LustreError: 322659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10209.254568] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10213.014269] LustreError: 321927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10213.050532] LustreError: 321927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10213.187875] LustreError: 321912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757807 with bad export cookie 13454542019423045861 [10213.206580] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10213.208595] LustreError: 321912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10214.377640] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10214.386678] Lustre: Skipped 1 previous similar message [10214.390094] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10218.121106] LustreError: 321926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10218.153887] LustreError: 321926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10218.181297] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [10219.827387] Lustre: server umount lustre-MDT0001 complete [10230.647135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10242.757662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10258.966086] LustreError: 323768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10259.000632] LustreError: 323768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10259.094689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10259.180161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10267.599024] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10272.762103] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10277.869718] LustreError: 325553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10277.907742] LustreError: 325553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10279.055658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10279.741891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10284.468977] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10285.037610] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10293.336408] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10295.011439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10301.554666] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10303.200777] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10310.128938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10310.141915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10310.176637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10310.194990] Lustre: Skipped 1 previous similar message [10315.026488] Lustre: server umount lustre-OST0000 complete [10320.357193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10320.371757] LustreError: Skipped 1 previous similar message [10320.383859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.399569] Lustre: Skipped 1 previous similar message [10320.406308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10320.413748] Lustre: Skipped 1 previous similar message [10325.335752] Lustre: server umount lustre-MDT0000 complete [10325.987486] LustreError: 325558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10326.018576] LustreError: 325558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10329.310574] LustreError: 321912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757923 with bad export cookie 13454542019423053610 [10329.325486] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10329.332108] LustreError: 321912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10330.082169] Lustre: server umount lustre-MDT0001 complete [10344.331624] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [10347.832420] Key type lgssc unregistered [10348.180910] LNet: 328232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10348.191051] LNetError: 328232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10349.224293] LNet: Removed LNI 192.168.202.118@tcp [10350.236639] Key type .llcrypt unregistered [10350.240088] Key type ._llcrypt unregistered [10379.982202] Key type ._llcrypt registered [10379.984991] Key type .llcrypt registered [10380.064878] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [10380.953694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10381.054039] alg: No test for adler32 (adler32-zlib) [10382.077909] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [10382.244432] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [10384.032367] Key type lgssc registered [10385.389806] Lustre: Echo OBD driver; http://www.lustre.org/ [10392.935946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10392.971089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10394.340067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10398.888636] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10403.811686] LustreError: 329650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10403.831838] LustreError: 329650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10407.697447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10408.170239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10412.668931] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10421.626466] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10428.451882] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10432.874233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10432.876877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10432.890587] Lustre: Skipped 1 previous similar message [10432.904018] Lustre: Skipped 1 previous similar message [10438.114123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10438.123083] Lustre: Skipped 1 previous similar message [10443.237154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10446.304342] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10446.540338] Lustre: server umount lustre-MDT0000 complete [10448.354253] LustreError: 330677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10448.382260] LustreError: 330677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10450.573453] LustreError: 329629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758045 with bad export cookie 14616660883495127808 [10450.589423] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10451.234343] Lustre: server umount lustre-MDT0001 complete [10467.697788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10467.958699] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10468.023027] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10468.251708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10473.020727] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10482.694248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10482.881329] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10482.903748] Lustre: 332242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10483.190087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10487.486292] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10495.941664] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10502.506925] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10513.408099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10513.838105] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10513.853927] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10513.864854] LustreError: 334129:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10513.875663] LustreError: 334129:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10513.879724] LustreError: 334129:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10513.943401] Lustre: server umount lustre-OST0000 complete [10513.946644] LustreError: 334129:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10526.833997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10527.097992] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10527.428451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10535.218973] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10537.150352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10544.422381] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10553.787216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10553.898439] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10554.020324] Lustre: lustre-OST0001: new disk, initializing [10554.028888] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10554.047758] Lustre: 335671:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10554.217899] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10560.100079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10560.107738] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10560.220768] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10561.140327] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10569.492840] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10575.863782] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10575.868167] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10575.877437] LustreError: Skipped 1 previous similar message [10575.903131] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10575.918088] Lustre: Skipped 2 previous similar messages [10579.280638] Lustre: server umount lustre-OST0001 complete [10580.972863] LustreError: 334597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10580.993509] LustreError: 334597:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10586.081860] LustreError: 334594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10586.085502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10586.116150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10586.117425] LustreError: 334594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10586.160944] Lustre: Skipped 2 previous similar messages [10591.224121] LustreError: 335974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10591.249846] LustreError: 335974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10596.323953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10596.333469] Lustre: Skipped 4 previous similar messages [10598.369161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10598.576384] Lustre: server umount lustre-OST0000 complete [10606.056681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10606.075263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10608.746964] Lustre: server umount lustre-MDT0000 complete [10611.683143] LustreError: 333310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10611.702755] LustreError: 333310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10612.960545] LustreError: 332236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758207 with bad export cookie 14616660883495128361 [10612.965799] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10612.973450] LustreError: 332236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10613.464779] Lustre: server umount lustre-MDT0001 complete [10625.482662] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [10629.406625] Key type lgssc unregistered [10629.855512] LNet: 337634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10629.873138] LNetError: 337634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10629.890210] LNet: Removed LNI 192.168.202.118@tcp [10631.317297] Key type .llcrypt unregistered [10631.319221] Key type ._llcrypt unregistered [10717.890498] Key type ._llcrypt registered [10717.898811] Key type .llcrypt registered [10718.131101] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [10736.760704] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [10738.112433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10738.166632] alg: No test for adler32 (adler32-zlib) [10739.355585] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [10739.612908] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [10741.304312] Key type lgssc registered [10742.719568] Lustre: Echo OBD driver; http://www.lustre.org/ [10783.638188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10783.682525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10785.089437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10785.116557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10785.192372] Lustre: lustre-MDT0000: new disk, initializing [10785.284596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10785.299905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10790.754496] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10803.231825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10803.292613] Lustre: 342577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10803.335620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10803.340232] Lustre: Skipped 1 previous similar message [10803.397366] Lustre: lustre-MDT0001: new disk, initializing [10803.481500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10803.553538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10803.566130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10808.928887] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10819.187494] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10825.726876] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10832.856585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10833.158589] Lustre: lustre-OST0000: new disk, initializing [10833.162899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10833.169710] Lustre: 344543:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10833.267580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10840.524479] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10840.677971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10840.701139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10840.911093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10851.501684] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10856.685206] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10857.000373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10862.293193] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10862.511629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10866.659992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10866.663627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10866.678321] LustreError: Skipped 1 previous similar message [10866.708621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10871.784389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10871.790758] Lustre: Skipped 1 previous similar message [10876.912828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10876.929116] Lustre: Skipped 1 previous similar message [10879.968455] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10880.370294] Lustre: server umount lustre-OST0000 complete [10890.722111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10890.737076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10890.755438] Lustre: Skipped 1 previous similar message [10890.759961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10890.764860] Lustre: Skipped 1 previous similar message [10892.263061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10893.712886] Lustre: server umount lustre-MDT0000 complete [10898.928888] LustreError: 342568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758493 with bad export cookie 16320494577503940938 [10898.963441] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10899.393695] Lustre: server umount lustre-MDT0001 complete [10909.555854] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:48:22 (1786758502) [10919.705385] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:48:32 (1786758512) [10931.109623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10931.964636] LustreError: 347301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10932.002693] LustreError: 347301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10932.150595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10937.329506] LustreError: 347302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10937.365421] LustreError: 347302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10939.330651] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10941.474847] LustreError: 347301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10946.530465] LustreError: 347302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10951.653120] LustreError: 347301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10952.077957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10952.701932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10958.894569] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10968.164573] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10974.567815] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10982.389829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10982.752546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10990.143607] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10998.237696] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11009.862862] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11011.781892] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11013.714776] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11015.982841] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11018.401675] Lustre: Failing over lustre-MDT0000 [11018.816124] Lustre: server umount lustre-MDT0000 complete [11019.233525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11019.245376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11019.268274] LustreError: 347301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11035.855076] LustreError: 347297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11035.882837] LustreError: 347297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [11039.200749] Lustre: 340798:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758617/real 1786758617] req@00000000127f97b8 x1873551906915584/t0(0) o400->MGC192.168.202.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786758633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11039.235661] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11039.502255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11049.458655] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe27e0b593acfb900 [11049.469802] Lustre: MGC192.168.202.118@tcp: Connection restored to 0@lo (at 0@lo) [11049.864342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11049.958945] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11051.136841] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11055.092428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11055.121230] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11055.191296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11056.254351] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11067.168382] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11069.444240] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11073.257646] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11075.743847] Lustre: Failing over lustre-MDT0000 [11076.048395] Lustre: server umount lustre-MDT0000 complete [11076.783064] LustreError: 347296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11076.808937] LustreError: 347296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [11080.677346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11080.684587] Lustre: Skipped 3 previous similar messages [11080.689058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11097.057613] Lustre: 340799:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758675/real 1786758675] req@00000000be7259a1 x1873551906937984/t0(0) o400->MGC192.168.202.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786758691 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11097.106383] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11097.358702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11107.757202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11107.833834] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11108.488731] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11112.956065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11112.971567] Lustre: Skipped 2 previous similar messages [11113.027985] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11113.097032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11113.820835] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11125.534390] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11127.364436] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11133.410262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11133.415296] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11133.444682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11133.468432] Lustre: Skipped 1 previous similar message [11133.473484] Lustre: Skipped 2 previous similar messages [11139.351123] Lustre: server umount lustre-MDT0000 complete [11143.655021] LustreError: 348031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11143.680748] LustreError: 348031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 29 previous similar messages [11143.820484] LustreError: 347283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758738 with bad export cookie 16320494577503943696 [11143.840014] LustreError: 347283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11143.843672] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11144.380062] Lustre: server umount lustre-MDT0001 complete [11159.812233] Lustre: server umount lustre-OST0000 complete [11181.628185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11182.556340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11189.269827] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11200.197643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11206.881817] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11216.415334] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11223.693468] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11232.689325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11241.795216] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11251.087358] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11252.977734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11261.421552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11261.441160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11261.465604] Lustre: Skipped 1 previous similar message [11261.474282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11261.481519] Lustre: Skipped 5 previous similar messages [11267.025186] Lustre: server umount lustre-OST0000 complete [11278.651390] Lustre: server umount lustre-MDT0000 complete [11278.819890] LustreError: 353673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.857610] LustreError: 353673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11284.671243] LustreError: 353656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758879 with bad export cookie 16320494577503944431 [11284.679803] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11284.686357] LustreError: 353656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11285.527598] Lustre: server umount lustre-MDT0001 complete [11302.120967] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [11306.154254] Key type lgssc unregistered [11306.750067] LNet: 357218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11306.765689] LNetError: 357218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11306.786777] LNet: Removed LNI 192.168.202.118@tcp [11308.169754] Key type .llcrypt unregistered [11308.171823] Key type ._llcrypt unregistered [11339.144521] Key type ._llcrypt registered [11339.146266] Key type .llcrypt registered [11339.327868] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [11340.699296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11341.039517] alg: No test for adler32 (adler32-zlib) [11342.200936] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [11342.583690] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [11344.298340] Key type lgssc registered [11345.726839] Lustre: Echo OBD driver; http://www.lustre.org/ [11353.356405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11353.378430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11354.950129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11360.551796] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11365.350224] LustreError: 358735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11365.372254] LustreError: 358735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11370.062308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11370.490984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11374.578133] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11382.689374] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11389.190112] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11396.750256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11397.080229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11400.195227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11404.908653] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11413.403990] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11426.785782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11426.797719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11426.813091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11429.857634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11429.859531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11429.889457] Lustre: Skipped 2 previous similar messages [11432.348726] Lustre: server umount lustre-MDT0000 complete [11434.987889] LustreError: 361228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11435.015873] LustreError: 361228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11436.124368] LustreError: 358716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759030 with bad export cookie 15925064854770437799 [11436.131292] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11436.364693] Lustre: server umount lustre-MDT0001 complete [11449.873176] Lustre: server umount lustre-OST0000 complete [11467.105055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11467.845401] LustreError: 362448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11468.043496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11473.381448] LustreError: 362449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11473.433580] LustreError: 362449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11475.856826] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11477.477892] LustreError: 362448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11484.335394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11484.865250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11490.837705] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11498.721677] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11504.704330] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11511.575676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11512.003991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11513.013971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11519.398157] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11528.902914] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11546.088339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11546.099885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11546.108194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11546.139960] Lustre: Skipped 2 previous similar messages [11550.014936] Lustre: server umount lustre-MDT0000 complete [11551.203258] LustreError: 363179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11551.222741] LustreError: 363179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11553.443243] LustreError: 362429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759147 with bad export cookie 15925064854770438912 [11553.447739] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11553.460748] LustreError: 362429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11554.186399] Lustre: server umount lustre-MDT0001 complete [11569.439544] Lustre: server umount lustre-OST0000 complete [11592.799437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11593.595818] LustreError: 366363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.757274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11598.173815] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11608.863234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11609.058797] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11609.071262] Lustre: Skipped 2 previous similar messages [11614.026635] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11622.715196] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11628.921985] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11636.669275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11637.127967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11637.140359] Lustre: Skipped 1 previous similar message [11639.217271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11643.667594] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11651.811629] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11658.721668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11658.732471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11658.735565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11658.747020] Lustre: Skipped 2 previous similar messages [11658.759362] Lustre: Skipped 1 previous similar message [11659.369352] Lustre: server umount lustre-OST0000 complete [11663.841558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11668.962674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11668.971607] Lustre: Skipped 3 previous similar messages [11669.680391] Lustre: server umount lustre-MDT0000 complete [11673.580697] LustreError: 367100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759268 with bad export cookie 15925064854770439927 [11673.596258] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11673.599274] LustreError: 367100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11674.036515] Lustre: server umount lustre-MDT0001 complete [11686.932217] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [11691.545783] Key type lgssc unregistered [11691.827748] LNet: 369918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11691.831712] LNetError: 369918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11691.847381] LNet: Removed LNI 192.168.202.118@tcp [11692.738950] Key type .llcrypt unregistered [11692.740828] Key type ._llcrypt unregistered [11721.165675] Key type ._llcrypt registered [11721.168206] Key type .llcrypt registered [11721.264438] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [11722.554580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11722.855909] alg: No test for adler32 (adler32-zlib) [11724.324361] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [11724.780623] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [11726.488687] Key type lgssc registered [11727.704106] Lustre: Echo OBD driver; http://www.lustre.org/ [11737.460553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11737.470428] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11737.474956] LustreError: 371361:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11737.487438] LustreError: 371361:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11737.496775] LustreError: 371361:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11737.502085] LustreError: 371361:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11737.509667] LustreError: 371361:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11745.231730] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11745.239388] LustreError: 371760:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11745.255511] LustreError: 371760:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11745.260556] LustreError: 371760:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11745.267480] LustreError: 371760:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11745.274346] LustreError: 371760:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11765.594823] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [11768.075624] Key type lgssc unregistered [11768.325316] LNet: 372766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11768.336565] LNetError: 372766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11768.370736] LNet: Removed LNI 192.168.202.118@tcp [11769.471357] Key type .llcrypt unregistered [11769.472941] Key type ._llcrypt unregistered [11791.841446] Key type ._llcrypt registered [11791.843042] Key type .llcrypt registered [11791.931582] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [11807.355486] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [11808.288038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11808.307920] alg: No test for adler32 (adler32-zlib) [11809.622867] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [11809.941779] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [11811.624230] Key type lgssc registered [11812.978825] Lustre: Echo OBD driver; http://www.lustre.org/ [11850.544355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11850.584099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11851.867954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11851.898436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11851.977573] Lustre: lustre-MDT0000: new disk, initializing [11852.032358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11852.042667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11856.708742] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11871.139617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11871.395381] Lustre: 375924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11871.513495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11871.518201] Lustre: Skipped 1 previous similar message [11871.595311] Lustre: lustre-MDT0001: new disk, initializing [11871.741927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11871.803039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11871.817413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11879.351357] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11889.429547] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11895.931203] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11903.378845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11904.062964] Lustre: lustre-OST0000: new disk, initializing [11904.073831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11904.086425] Lustre: 377896:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11904.218071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11909.734406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11909.750784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11909.932631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11913.375932] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11924.632995] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11932.148280] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11932.617500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11939.823777] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11940.324441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11945.954519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11945.960268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11946.005369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11949.222377] Lustre: server umount lustre-OST0000 complete [11956.198650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11956.207115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11956.213897] Lustre: Skipped 2 previous similar messages [11956.238147] Lustre: Skipped 2 previous similar messages [11960.403339] Lustre: server umount lustre-MDT0000 complete [11964.730800] LustreError: 375915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759559 with bad export cookie 9613749298458613127 [11964.735530] LustreError: 375915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11964.739423] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11965.523463] Lustre: server umount lustre-MDT0001 complete [11977.030861] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11979.068362] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:06:11 (1786759571) [11985.930881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11986.581547] LustreError: 380358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11986.594049] LustreError: 380358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11986.655716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11992.042359] LustreError: 380359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11992.690911] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11997.173818] LustreError: 380358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12002.277584] LustreError: 380359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12002.496088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12003.005848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12008.340942] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12017.561545] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12027.493850] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12037.745280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12038.232929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12045.850187] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12053.979761] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12062.549805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12062.826575] Lustre: lustre-OST0001: new disk, initializing [12062.829781] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12062.839637] Lustre: 383200:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12062.918628] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12067.574872] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12069.010716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12069.034331] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12069.169160] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12080.339083] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12087.497172] Lustre: lustre-OST0001: Not available for connect from 192.168.202.18@tcp (stopping) [12089.825744] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12089.827653] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12089.841608] LustreError: Skipped 1 previous similar message [12089.850196] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12089.859765] Lustre: Skipped 1 previous similar message [12089.892865] Lustre: Skipped 1 previous similar message [12091.557460] Lustre: server umount lustre-OST0001 complete [12092.588713] LustreError: 382186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12105.217265] LustreError: 383636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12105.255480] LustreError: 383636:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [12113.379885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12113.393381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12113.417347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12118.555962] Lustre: server umount lustre-OST0000 complete [12125.671493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12125.672052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12125.674613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12125.674620] Lustre: Skipped 3 previous similar messages [12125.685625] Lustre: Skipped 2 previous similar messages [12125.693071] LustreError: Skipped 1 previous similar message [12129.349817] Lustre: server umount lustre-MDT0000 complete [12130.788489] LustreError: 380354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12130.797237] LustreError: 380354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12134.076216] LustreError: 380339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759728 with bad export cookie 9613749298458614051 [12134.080110] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12134.082812] LustreError: 380339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12140.794201] Lustre: server umount lustre-MDT0001 complete [12152.395098] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12154.914418] Key type lgssc unregistered [12155.247750] LNet: 385192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12155.256409] LNetError: 385192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12156.328133] LNet: Removed LNI 192.168.202.118@tcp [12157.379164] Key type .llcrypt unregistered [12157.384891] Key type ._llcrypt unregistered [12211.099537] Key type ._llcrypt registered [12211.101865] Key type .llcrypt registered [12211.230166] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12212.735818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12212.781362] alg: No test for adler32 (adler32-zlib) [12213.829722] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [12214.267231] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12216.121325] Key type lgssc registered [12217.716989] Lustre: Echo OBD driver; http://www.lustre.org/ [12226.780715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12226.815435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12228.121609] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12228.152515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12228.393274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12233.773206] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12243.102872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12243.281656] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12243.317673] Lustre: 388004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12243.690951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12249.397861] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12258.391625] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12265.148120] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12269.537709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12269.558572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12269.590113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12270.562808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12270.577599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12273.925499] Lustre: server umount lustre-MDT0000 complete [12278.299052] LustreError: 387996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759872 with bad export cookie 8228522539510122778 [12278.309953] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12278.777858] Lustre: server umount lustre-MDT0001 complete [12289.928760] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:11:22 (1786759882) [12297.474557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12298.017191] LustreError: 390517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12298.031703] LustreError: 390517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12298.089371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12302.640197] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12303.337472] LustreError: 390518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12308.449642] LustreError: 390517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12311.726337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12312.107091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12317.456117] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12325.129938] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12331.177652] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12338.995262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12339.192294] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12339.361226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12346.341407] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12353.744877] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12360.588710] Lustre: Permanently deactivating lustre-OST0000 [12360.624201] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12367.420165] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12369.618959] LustreError: 393505:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12369.629663] LustreError: 393505:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12369.633990] LustreError: 393505:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12372.415822] LustreError: 393554:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12372.434660] LustreError: 393554:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12380.608770] Lustre: server umount lustre-OST0000 complete [12383.298946] LustreError: 393804:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12383.307465] LustreError: 393804:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12388.323342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12388.331440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12388.340670] Lustre: Skipped 1 previous similar message [12388.349761] Lustre: Skipped 2 previous similar messages [12393.444292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12393.455704] Lustre: Skipped 1 previous similar message [12394.612356] Lustre: server umount lustre-MDT0000 complete [12398.566248] LustreError: 390512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12398.591487] LustreError: 390512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12399.981747] LustreError: 390499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759994 with bad export cookie 8228522539510123359 [12399.983492] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12399.988956] LustreError: 390499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12410.640978] Lustre: server umount lustre-MDT0001 complete [12423.032376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12430.803670] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12434.198355] LustreError: 394964:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12442.181037] Lustre: 395303:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.118@tcp [12444.795176] Lustre: server umount MGS complete [12457.266494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12457.689401] LustreError: 396027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12457.707377] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12457.711733] Lustre: Skipped 1 previous similar message [12457.757120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12462.043676] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12468.193048] LustreError: 396028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12468.210954] LustreError: 396028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12470.155623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12470.385772] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12475.126964] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12482.542517] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12488.188979] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12490.085579] Lustre: Permanently reactivating lustre-OST0000 [12490.111644] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12490.130266] Lustre: Skipped 2 previous similar messages [12517.856376] LustreError: 397811:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12526.457119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12526.910218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12526.913086] Lustre: Skipped 1 previous similar message [12533.990754] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12543.554204] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12551.027823] Lustre: server umount lustre-OST0000 complete [12557.794080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12557.807245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12557.828742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12565.071753] Lustre: server umount lustre-MDT0000 complete [12569.057766] LustreError: 396023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12569.062183] Lustre: 387328:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786760147/real 1786760147] req@00000000b03174f6 x1873553452534656/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786760163 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12569.075971] LustreError: 396023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12569.112259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12569.117771] Lustre: Skipped 1 previous similar message [12569.390986] LustreError: 396007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760163 with bad export cookie 8228522539510124409 [12569.395144] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12569.402396] LustreError: 396007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12569.941975] Lustre: server umount lustre-MDT0001 complete [12583.162394] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12585.797441] Key type lgssc unregistered [12586.205662] LNet: 400351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12586.215172] LNetError: 400351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12587.243966] LNet: Removed LNI 192.168.202.118@tcp [12588.423690] Key type .llcrypt unregistered [12588.427907] Key type ._llcrypt unregistered [12609.256922] Key type ._llcrypt registered [12609.258575] Key type .llcrypt registered [12609.347927] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [12631.749597] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12632.780711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12632.822719] alg: No test for adler32 (adler32-zlib) [12633.855418] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [12634.030337] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12635.696155] Key type lgssc registered [12636.585028] Lustre: Echo OBD driver; http://www.lustre.org/ [12676.437436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12676.452084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12677.803219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12677.844851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12677.938959] Lustre: lustre-MDT0000: new disk, initializing [12678.054511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12678.070291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12682.241541] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12693.001701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12693.106807] Lustre: 403525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12693.159366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12693.164588] Lustre: Skipped 1 previous similar message [12693.235163] Lustre: lustre-MDT0001: new disk, initializing [12693.328524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12693.376070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12693.388967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12698.289276] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12708.177662] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12713.942093] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12721.449108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12721.819230] Lustre: lustre-OST0000: new disk, initializing [12721.822905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12721.830951] Lustre: 405496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12721.984506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12728.618421] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12728.893101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12728.905915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12729.049979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12738.255291] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12742.418117] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12742.619442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12747.797837] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12748.152322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12754.929433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12754.931627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12754.939631] LustreError: Skipped 1 previous similar message [12754.976549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12754.991782] Lustre: Skipped 1 previous similar message [12757.696636] Lustre: server umount lustre-OST0000 complete [12765.156433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12765.171153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12765.183369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12765.195675] Lustre: Skipped 2 previous similar messages [12765.210493] Lustre: Skipped 1 previous similar message [12769.424377] Lustre: server umount lustre-MDT0000 complete [12773.707436] LustreError: 403517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760368 with bad export cookie 6487478483726256370 [12773.716922] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12773.718510] LustreError: 403517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12774.156353] Lustre: server umount lustre-MDT0001 complete [12785.381565] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12788.520822] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:19:40 (1786760380) [12796.497391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12796.982718] LustreError: 407960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12796.994529] LustreError: 407960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12797.076515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12802.531516] LustreError: 407961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12802.556275] LustreError: 407961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12803.371540] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12806.626563] LustreError: 407960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12811.749567] LustreError: 407961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12813.480969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12814.066471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12819.639592] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12828.956662] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12835.696855] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12846.348543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12846.710225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12857.459853] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12866.616048] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12869.040100] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12872.224363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12879.842677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12879.852080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12879.864538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12882.913718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12882.916745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12882.924856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12882.963498] Lustre: Skipped 1 previous similar message [12885.408636] Lustre: server umount lustre-OST0000 complete [12891.106663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12891.114343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12891.127802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12897.077077] Lustre: server umount lustre-MDT0000 complete [12898.272829] LustreError: 407957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12898.280396] LustreError: 407957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12901.862611] LustreError: 407940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760496 with bad export cookie 6487478483726257294 [12901.870826] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12901.893723] LustreError: 407940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12902.424186] Lustre: server umount lustre-MDT0001 complete [12920.582272] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12924.548189] Key type lgssc unregistered [12925.120737] LNet: 411715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12925.131624] LNetError: 411715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12925.155784] LNet: Removed LNI 192.168.202.118@tcp [12926.434780] Key type .llcrypt unregistered [12926.439145] Key type ._llcrypt unregistered [12957.687597] Key type ._llcrypt registered [12957.692572] Key type .llcrypt registered [12957.817984] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12958.365194] Key type .llcrypt unregistered [12958.367456] Key type ._llcrypt unregistered [12977.660267] Key type ._llcrypt registered [12977.663240] Key type .llcrypt registered [12977.832719] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12979.396567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12979.458797] alg: No test for adler32 (adler32-zlib) [12980.810994] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [12981.200881] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12982.944192] Key type lgssc registered [12984.320715] Lustre: Echo OBD driver; http://www.lustre.org/ [12992.793799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12992.823078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12994.288519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12999.093349] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13003.768348] LustreError: 413931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13003.804121] LustreError: 413931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13008.488458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13008.872260] LustreError: 413932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13008.974705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13013.990133] LustreError: 413928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13014.019628] LustreError: 413928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13016.992726] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13019.126116] LustreError: 413931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13019.141281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13019.150095] LustreError: 413931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13024.228107] LustreError: 413932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13024.265044] LustreError: 413932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13025.145348] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13033.023400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13033.301144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13037.248165] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13048.812172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13048.813706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13048.825041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13048.825240] Lustre: Skipped 1 previous similar message [13053.930503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13053.940262] Lustre: Skipped 1 previous similar message [13054.237503] Lustre: server umount lustre-OST0000 complete [13060.666887] Lustre: server umount lustre-MDT0000 complete [13064.167131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13064.173570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13064.190102] LustreError: 413931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13064.190113] LustreError: 413931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13064.247733] Lustre: Skipped 1 previous similar message [13065.942702] LustreError: 413912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760660 with bad export cookie 17689323403326531319 [13065.950529] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13066.270186] Lustre: server umount lustre-MDT0001 complete [13079.939773] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13083.167873] Key type lgssc unregistered [13083.484890] LNet: 417084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13083.490790] LNetError: 417084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13084.522926] LNet: Removed LNI 192.168.202.118@tcp [13085.427394] Key type .llcrypt unregistered [13085.431497] Key type ._llcrypt unregistered [13117.214529] Key type ._llcrypt registered [13117.216167] Key type .llcrypt registered [13117.428675] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13119.011238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13119.632734] alg: No test for adler32 (adler32-zlib) [13121.255458] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [13122.135755] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [13123.904197] Key type lgssc registered [13125.429432] Lustre: Echo OBD driver; http://www.lustre.org/ [13134.981624] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13135.013477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13150.624406] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13155.744299] LustreError: 418478:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.118@tcp: failed processing log, type 4: rc = -110 [13182.433086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13190.119389] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13197.623774] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13204.402770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13204.856402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13204.939737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13209.917861] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13210.086087] LustreError: 418503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13210.098830] LustreError: 418503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13215.207674] LustreError: 418503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13217.254214] LustreError: 418504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13217.280617] LustreError: 418504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13218.796393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13219.383654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13224.309460] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13232.031537] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13237.928448] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13244.897982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13244.916843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13244.922154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13244.942371] Lustre: Skipped 1 previous similar message [13246.771743] Lustre: server umount lustre-OST0000 complete [13253.098247] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13253.102597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13253.119800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13253.128214] Lustre: Skipped 1 previous similar message [13255.153317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13255.157643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13255.189972] Lustre: Skipped 1 previous similar message [13257.405521] Lustre: server umount lustre-MDT0000 complete [13260.273392] LustreError: 420252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13260.291141] LustreError: 420252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13261.785633] LustreError: 418486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760856 with bad export cookie 15264818063803601806 [13261.821529] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13262.249799] Lustre: server umount lustre-MDT0001 complete [13273.096221] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13275.927796] Key type lgssc unregistered [13276.215472] LNet: 422052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13276.228482] LNetError: 422052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13276.250744] LNet: Removed LNI 192.168.202.118@tcp [13277.314314] Key type .llcrypt unregistered [13277.315750] Key type ._llcrypt unregistered [13304.440459] Key type ._llcrypt registered [13304.446845] Key type .llcrypt registered [13304.598060] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13305.881397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13306.343104] alg: No test for adler32 (adler32-zlib) [13307.606791] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [13308.013479] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [13309.800244] Key type lgssc registered [13311.031985] Lustre: Echo OBD driver; http://www.lustre.org/ [13319.390626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13319.409639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13321.111861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13327.316530] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13331.426420] LustreError: 423468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13331.457788] LustreError: 423468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13336.545201] LustreError: 423469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13336.977594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13337.645806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13342.890210] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13351.895519] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13357.624029] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13365.693914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13366.262790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13367.293108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13374.838551] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13383.909456] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13391.780607] Lustre: server umount lustre-MDT0000 complete [13392.059356] LustreError: 423462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13392.868667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13392.876450] Lustre: Skipped 1 previous similar message [13394.897705] LustreError: 423449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.18@tcp arrived at 1786760989 with bad export cookie 10382052214018354753 [13394.914990] LustreError: 423449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13397.337196] LustreError: 423450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760991 with bad export cookie 10382052214018353857 [13397.356726] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13397.997286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13398.008272] LustreError: 423469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13398.009439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13398.017485] Lustre: Skipped 1 previous similar message [13398.027683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13398.053165] LustreError: 423469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13403.105802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13403.105950] LustreError: 424438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13403.136618] LustreError: 424438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13403.745608] Lustre: server umount lustre-OST0000 complete [13412.399565] Lustre: server umount lustre-MDT0001 complete [13423.861935] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13427.861279] Key type lgssc unregistered [13428.111319] LNet: 427163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13428.123676] LNetError: 427163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13428.148299] LNet: Removed LNI 192.168.202.118@tcp [13429.222155] Key type .llcrypt unregistered [13429.225071] Key type ._llcrypt unregistered [13458.315429] Key type ._llcrypt registered [13458.328138] Key type .llcrypt registered [13458.574116] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13461.217857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13461.260691] alg: No test for adler32 (adler32-zlib) [13462.434586] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [13462.674929] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [13464.496360] Key type lgssc registered [13465.831914] Lustre: Echo OBD driver; http://www.lustre.org/ [13478.648979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13478.688917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13481.125272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13488.296377] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13490.659890] LustreError: 428579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13490.697487] LustreError: 428579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13495.782431] LustreError: 428580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13499.177424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13499.810465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13505.782842] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13514.443987] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13522.353715] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13530.674796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13531.140215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13538.668555] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13547.753821] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13550.593304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13559.969578] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [13560.803878] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13560.826772] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13560.834843] Lustre: Skipped 1 previous similar message [13563.781231] Lustre: server umount lustre-MDT0001 complete [13565.067453] LustreError: 428576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13571.041201] LustreError: 428579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13571.044333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13571.046238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13571.046247] Lustre: Skipped 1 previous similar message [13571.047975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13571.061299] LustreError: 428579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13576.168696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13576.170782] LustreError: 429309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13577.050569] Lustre: server umount lustre-OST0000 complete [13581.487238] Lustre: server umount lustre-MDT0000 complete [13597.510638] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13600.926868] Key type lgssc unregistered [13601.279235] LNet: 432272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13601.304638] LNetError: 432272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13601.325386] LNet: Removed LNI 192.168.202.118@tcp [13602.528265] Key type .llcrypt unregistered [13602.531184] Key type ._llcrypt unregistered [13628.677809] Key type ._llcrypt registered [13628.682541] Key type .llcrypt registered [13628.836484] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [13646.315684] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13648.032894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13648.066584] alg: No test for adler32 (adler32-zlib) [13649.277432] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [13649.501108] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [13651.296355] Key type lgssc registered [13652.797128] Lustre: Echo OBD driver; http://www.lustre.org/ [13698.350684] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13717.952458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13717.997049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13719.310982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13719.416348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13719.558966] Lustre: lustre-MDT0000: new disk, initializing [13719.732350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13719.756934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13725.788890] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13745.777486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13746.127426] Lustre: lustre-OST0000: new disk, initializing [13746.131651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13746.139661] Lustre: Skipped 1 previous similar message [13746.143924] Lustre: 437313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13746.235530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13754.301254] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13756.477036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13756.489152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13756.533107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13765.210101] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13782.718592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13782.863603] Lustre: 436291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13782.955667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13783.007777] Lustre: lustre-MDT0001: new disk, initializing [13783.100757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13783.131921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13783.142661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13788.254634] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13798.751871] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13798.964552] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13804.513656] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13804.818798] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13812.200256] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13812.202455] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13812.211123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13812.226921] Lustre: Skipped 1 previous similar message [13812.241627] Lustre: Skipped 1 previous similar message [13814.957568] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [13816.439937] Lustre: server umount lustre-MDT0001 complete [13822.446477] LustreError: 438025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13822.485236] LustreError: 438025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13827.559620] LustreError: 436298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13827.575506] LustreError: 436298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13832.689989] LustreError: 436299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13832.707893] LustreError: 436299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13837.793418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13837.801532] LustreError: 436303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13837.827726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13841.402347] Lustre: server umount lustre-OST0000 complete [13842.915134] LustreError: 438655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13860.320572] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13860.898701] Lustre: server umount lustre-MDT0000 complete [13876.048580] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13879.775597] Key type lgssc unregistered [13880.295780] LNet: 441069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13880.311891] LNetError: 441069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13880.325939] LNet: Removed LNI 192.168.202.118@tcp [13881.120238] Key type .llcrypt unregistered [13881.122047] Key type ._llcrypt unregistered [13928.380352] Key type ._llcrypt registered [13928.383452] Key type .llcrypt registered [13928.560974] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13928.897809] Key type .llcrypt unregistered [13928.899895] Key type ._llcrypt unregistered [13945.355655] Key type ._llcrypt registered [13945.357910] Key type .llcrypt registered [13945.461689] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13946.403421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13946.419082] alg: No test for adler32 (adler32-zlib) [13947.600540] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [13947.831916] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [13949.472233] Key type lgssc registered [13950.828699] Lustre: Echo OBD driver; http://www.lustre.org/ [13978.590112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13978.618286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13979.814287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13979.849602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13979.924468] Lustre: lustre-MDT0000: new disk, initializing [13979.983979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13979.992055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13983.627863] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13994.956599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13995.100620] Lustre: 444452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13995.151508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13995.154860] Lustre: Skipped 1 previous similar message [13995.219907] Lustre: lustre-MDT0001: new disk, initializing [13995.340936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13995.381554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13995.391706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14000.837726] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14011.408740] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14017.990714] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14027.014639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14027.493184] Lustre: lustre-OST0000: new disk, initializing [14027.497209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14027.503524] Lustre: 446422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14027.688458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14036.566385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14036.576825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14036.807989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14036.953260] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14049.946404] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14062.057192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14062.068408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14062.081908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14062.576677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14062.602104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14062.613851] Lustre: Skipped 1 previous similar message [14064.956410] Lustre: server umount lustre-MDT0000 complete [14070.075888] LustreError: 446421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761664 with bad export cookie 16657617211953147713 [14070.081136] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14070.091199] LustreError: 446421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14070.511541] Lustre: server umount lustre-MDT0001 complete [14085.432808] Lustre: server umount lustre-OST0000 complete [14096.803856] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:41:30 (1786761690) [14105.779165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14106.416436] LustreError: 448450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14106.438826] LustreError: 448450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14106.559425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14111.729898] LustreError: 448451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14112.376911] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14116.848262] LustreError: 448450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14121.958619] LustreError: 448451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14122.047443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14122.526386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14126.611535] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14133.547430] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14139.597185] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14160.619491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14160.847187] Lustre: Found index 0 for lustre-OST0000, updating log [14161.047696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14167.663768] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14176.049122] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14177.519846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14188.004967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14188.011528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14188.030588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14188.032663] Lustre: Skipped 2 previous similar messages [14188.063704] LustreError: Skipped 1 previous similar message [14193.151595] Lustre: server umount lustre-OST0000 complete [14210.536579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14210.551879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14210.566142] Lustre: Skipped 1 previous similar message [14210.595060] Lustre: Skipped 4 previous similar messages [14214.702712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14214.712742] Lustre: Skipped 1 previous similar message [14216.443736] Lustre: server umount lustre-MDT0000 complete [14220.769978] LustreError: 449668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14220.795097] LustreError: 449668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14221.053096] LustreError: 448432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761815 with bad export cookie 16657617211953149148 [14221.055302] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14221.062108] LustreError: 448432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14221.498774] Lustre: server umount lustre-MDT0001 complete [14223.458350] LNet: 1 local NIs in recovery (showing 1): 192.168.202.118@tcp [14231.558443] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:43:44 (1786761824) [14857.840711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14858.331543] LustreError: 454866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14858.432147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14863.320392] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14863.850334] LustreError: 454867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14867.940486] LustreError: 454866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14872.808547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14873.211134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14878.654891] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14886.646456] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14892.825638] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14913.767217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14913.900980] Lustre: Found index 0 for lustre-OST0000, updating log [14914.084577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14914.656125] LNet: 1 local NIs in recovery (showing 1): 192.168.202.118@tcp [14922.158655] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14932.790585] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14936.554463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14936.560834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14936.584419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14936.597677] Lustre: Skipped 2 previous similar messages [14942.989180] Lustre: server umount lustre-OST0000 complete [14947.453907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14947.483401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14947.495395] Lustre: Skipped 1 previous similar message [14949.539409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14949.548192] Lustre: Skipped 3 previous similar messages [14953.635115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14953.645898] Lustre: Skipped 7 previous similar messages [14953.849698] Lustre: server umount lustre-MDT0000 complete [14954.658809] LustreError: 454866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14954.700050] LustreError: 454866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14957.885282] LustreError: 454849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762552 with bad export cookie 16657617211953150226 [14957.893196] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14958.179982] Lustre: server umount lustre-MDT0001 complete [14975.078517] LNet: 1 local NIs in recovery (showing 1): 192.168.202.118@tcp [14994.080605] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [15004.060299] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [15040.774407] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:57:14 (1786762634) [15061.647281] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [15064.789867] Key type lgssc unregistered [15065.122248] LNet: 461486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15065.140409] LNetError: 461486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15176.808630] LNet: Removed LNI 192.168.202.118@tcp [15178.643228] Key type .llcrypt unregistered [15178.646106] Key type ._llcrypt unregistered [15196.727317] Key type ._llcrypt registered [15196.729351] Key type .llcrypt registered [15196.890508] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [15198.464367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15198.798091] alg: No test for adler32 (adler32-zlib) [15200.343857] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [15200.868506] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [15202.656389] Key type lgssc registered [15204.371863] Lustre: Echo OBD driver; http://www.lustre.org/ [15209.562058] LNet: Added LNI 192.168.202.118@tcp100 [8/256/0/180] [15210.634482] LNet: Added LNI 192.168.202.118@tcp101 [8/256/0/180] [15211.803481] LNet: Added LNI 192.168.202.118@tcp102 [8/256/0/180] [15213.123751] LNet: Added LNI 192.168.202.118@tcp103 [8/256/0/180] [15214.047299] LNet: Added LNI 192.168.202.118@tcp104 [8/256/0/180] [15215.070041] LNet: Added LNI 192.168.202.118@tcp105 [8/256/0/180] [15216.114526] LNet: Added LNI 192.168.202.118@tcp106 [8/256/0/180] [15217.543626] LNet: Added LNI 192.168.202.118@tcp107 [8/256/0/180] [15218.794778] LNet: Added LNI 192.168.202.118@tcp108 [8/256/0/180] [15219.925906] LNet: Added LNI 192.168.202.118@tcp109 [8/256/0/180] [15221.116723] LNet: Added LNI 192.168.202.118@tcp110 [8/256/0/180] [15222.316693] LNet: Added LNI 192.168.202.118@tcp111 [8/256/0/180] [15223.166734] LNet: Added LNI 192.168.202.118@tcp112 [8/256/0/180] [15224.195914] LNet: Added LNI 192.168.202.118@tcp113 [8/256/0/180] [15225.334255] LNet: Added LNI 192.168.202.118@tcp114 [8/256/0/180] [15226.419460] LNet: Added LNI 192.168.202.118@tcp115 [8/256/0/180] [15227.508489] LNet: Added LNI 192.168.202.118@tcp116 [8/256/0/180] [15228.583437] LNet: Added LNI 192.168.202.118@tcp117 [8/256/0/180] [15229.646564] LNet: Added LNI 192.168.202.118@tcp118 [8/256/0/180] [15230.452126] LNet: Added LNI 192.168.202.118@tcp119 [8/256/0/180] [15231.690279] LNet: Added LNI 192.168.202.118@tcp120 [8/256/0/180] [15232.986170] LNet: Added LNI 192.168.202.118@tcp121 [8/256/0/180] [15233.995926] LNet: Added LNI 192.168.202.118@tcp122 [8/256/0/180] [15235.009278] LNet: Added LNI 192.168.202.118@tcp123 [8/256/0/180] [15236.328270] LNet: Added LNI 192.168.202.118@tcp124 [8/256/0/180] [15237.253340] LNet: Added LNI 192.168.202.118@tcp125 [8/256/0/180] [15238.626268] LNet: Added LNI 192.168.202.118@tcp126 [8/256/0/180] [15239.494502] LNet: Added LNI 192.168.202.118@tcp127 [8/256/0/180] [15240.448878] LNet: Added LNI 192.168.202.118@tcp128 [8/256/0/180] [15241.333942] LNet: Added LNI 192.168.202.118@tcp129 [8/256/0/180] [15242.203778] LNet: Added LNI 192.168.202.118@tcp130 [8/256/0/180] [15243.057880] LNet: Added LNI 192.168.202.118@tcp131 [8/256/0/180] [15244.028494] LNet: Added LNI 192.168.202.118@tcp132 [8/256/0/180] [15245.115544] LNet: Added LNI 192.168.202.118@tcp133 [8/256/0/180] [15246.056057] LNet: Added LNI 192.168.202.118@tcp134 [8/256/0/180] [15247.106672] LNet: Added LNI 192.168.202.118@tcp135 [8/256/0/180] [15254.845937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15254.875933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15256.066323] Lustre: 464520:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [15256.134076] LNetError: 464542:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [15256.152668] LNetError: Protocol error connecting to 192.168.202.118@tcp123 at host 192.168.202.118:988: is it running a compatible version of Lustre? [15256.162787] LNetError: Skipped 4 previous similar messages [15256.169376] LNet: 1 local NIs in recovery (showing 1): 192.168.202.118@tcp126 [15256.276476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15256.297673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15256.376489] Lustre: lustre-MDT0000: new disk, initializing [15256.453725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15256.470853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15257.187752] LNetError: 464548:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [15257.199635] LNetError: 464548:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [15257.211810] LNetError: Protocol error connecting to 192.168.202.118@tcp109 at host 192.168.202.118:988: is it running a compatible version of Lustre? [15257.229798] LNetError: Skipped 2 previous similar messages [15259.234660] LNetError: 461978:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [15259.248137] LNetError: 461978:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [15259.261481] LNetError: Protocol error connecting to 192.168.202.118@tcp123 at host 192.168.202.118:988: is it running a compatible version of Lustre? [15260.131813] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15263.337534] LNetError: 461979:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [15263.379099] LNetError: Protocol error connecting to 192.168.202.118@tcp123 at host 192.168.202.118:988: is it running a compatible version of Lustre? [15271.523353] LNetError: 464546:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [15271.536060] LNetError: Protocol error connecting to 192.168.202.118@tcp123 at host 192.168.202.118:988: is it running a compatible version of Lustre? [15271.908212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15272.077861] Lustre: 464537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15272.146847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15272.150957] Lustre: Skipped 1 previous similar message [15272.238748] Lustre: lustre-MDT0001: new disk, initializing [15272.285618] Lustre: 465351:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [15272.295560] Lustre: 465351:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15272.466920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15272.508560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15272.526372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15277.494621] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15277.633977] Lustre: 465617:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [15277.648567] Lustre: 465617:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15287.240842] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15287.919538] LNetError: 464540:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [15287.935496] LNetError: Protocol error connecting to 192.168.202.118@tcp124 at host 192.168.202.118:988: is it running a compatible version of Lustre? [15294.481914] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15301.754417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15302.086742] Lustre: lustre-OST0000: new disk, initializing [15302.090453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15302.098885] Lustre: 466525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15302.199797] Lustre: 466530:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [15302.206714] Lustre: 466530:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15302.272764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15304.290257] LNetError: 464541:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [15304.304400] LNetError: 464541:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [15304.322743] LNetError: Protocol error connecting to 192.168.202.118@tcp125 at host 192.168.202.118:988: is it running a compatible version of Lustre? [15304.343888] LNetError: Skipped 7 previous similar messages [15308.897155] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15310.365420] Lustre: 466949:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [15310.376090] Lustre: 466949:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15310.448765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15310.467066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15310.598557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15319.594131] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15331.308484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15331.309658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15331.320372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15331.325956] LustreError: Skipped 1 previous similar message [15331.338115] Lustre: Skipped 1 previous similar message [15334.028025] Lustre: server umount lustre-OST0000 complete [15338.993718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15339.007383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15339.037390] Lustre: Skipped 1 previous similar message [15339.053058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15341.153434] LNetError: 461976:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [15341.200352] LNetError: 461976:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 14 previous similar messages [15341.218045] LNetError: Protocol error connecting to 192.168.202.118@tcp109 at host 192.168.202.118:988: is it running a compatible version of Lustre? [15341.236800] LNetError: Skipped 14 previous similar messages [15341.544746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15341.569217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15341.591198] Lustre: Skipped 1 previous similar message [15345.052031] Lustre: server umount lustre-MDT0000 complete [15349.378632] LustreError: 464530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762943 with bad export cookie 9291594987381998361 [15349.380919] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15349.394892] LustreError: 464530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15349.976471] Lustre: server umount lustre-MDT0001 complete [15361.994119] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [15365.552445] Key type lgssc unregistered [15365.957472] LNet: 468443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15365.963953] LNetError: 468443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15365.973398] LNet: Removed LNI 192.168.202.118@tcp [15367.009840] LNet: Removed LNI 192.168.202.118@tcp100 [15367.011837] LNetError: 468443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15367.021752] LNetError: 468443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15367.030250] LNet: Removed LNI 192.168.202.118@tcp101 [15368.096899] LNet: Removed LNI 192.168.202.118@tcp102 [15368.101858] LNetError: 468443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15368.109957] LNetError: 468443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15368.123550] LNet: Removed LNI 192.168.202.118@tcp103 [15369.184284] LNet: Removed LNI 192.168.202.118@tcp104 [15370.211074] LNet: Removed LNI 192.168.202.118@tcp105 [15370.216366] LNetError: 468443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15370.226315] LNetError: 468443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15370.234080] LNet: Removed LNI 192.168.202.118@tcp106 [15371.296379] LNet: Removed LNI 192.168.202.118@tcp107 [15372.320207] LNet: Removed LNI 192.168.202.118@tcp108 [15373.344619] LNet: Removed LNI 192.168.202.118@tcp109 [15374.368160] LNet: Removed LNI 192.168.202.118@tcp110 [15374.391080] LNetError: 468443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15374.402644] LNetError: 468443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15374.419196] LNet: Removed LNI 192.168.202.118@tcp111 [15375.456676] LNet: Removed LNI 192.168.202.118@tcp112 [15376.480310] LNet: Removed LNI 192.168.202.118@tcp113 [15377.504337] LNet: Removed LNI 192.168.202.118@tcp114 [15378.528764] LNet: Removed LNI 192.168.202.118@tcp115 [15379.552172] LNet: Removed LNI 192.168.202.118@tcp116 [15380.576182] LNet: Removed LNI 192.168.202.118@tcp117 [15381.601043] LNet: Removed LNI 192.168.202.118@tcp118 [15382.624669] LNet: Removed LNI 192.168.202.118@tcp119 [15382.634758] LNetError: 468443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15382.642101] LNetError: 468443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15382.651069] LNet: Removed LNI 192.168.202.118@tcp120 [15383.712746] LNet: Removed LNI 192.168.202.118@tcp121 [15384.736205] LNet: Removed LNI 192.168.202.118@tcp122 [15385.760462] LNet: Removed LNI 192.168.202.118@tcp123 [15386.784381] LNet: Removed LNI 192.168.202.118@tcp124 [15387.808216] LNet: Removed LNI 192.168.202.118@tcp125 [15388.832210] LNet: Removed LNI 192.168.202.118@tcp126 [15389.856409] LNet: Removed LNI 192.168.202.118@tcp127 [15390.880164] LNet: Removed LNI 192.168.202.118@tcp128 [15391.904564] LNet: Removed LNI 192.168.202.118@tcp129 [15391.910151] LNet: Removed LNI 192.168.202.118@tcp130 [15391.911700] LNet: Removed LNI 192.168.202.118@tcp131 [15391.924993] LNet: Removed LNI 192.168.202.118@tcp132 [15391.933208] LNet: Removed LNI 192.168.202.118@tcp133 [15391.940960] LNet: Removed LNI 192.168.202.118@tcp134 [15391.961636] LNet: Removed LNI 192.168.202.118@tcp135 [15392.827316] Key type .llcrypt unregistered [15392.832855] Key type ._llcrypt unregistered [15421.861323] Key type ._llcrypt registered [15421.863679] Key type .llcrypt registered [15422.030516] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [15439.104180] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [15441.259443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15441.329223] alg: No test for adler32 (adler32-zlib) [15442.632636] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [15443.038838] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [15444.824249] Key type lgssc registered [15446.412839] Lustre: Echo OBD driver; http://www.lustre.org/ [15490.029345] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:04:43 (1786763083) [15513.591969] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [15517.208787] Key type lgssc unregistered [15517.559022] LNet: 472538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15517.570070] LNetError: 472538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15517.592182] LNet: Removed LNI 192.168.202.118@tcp [15518.879457] Key type .llcrypt unregistered [15518.882989] Key type ._llcrypt unregistered [15538.443900] Key type ._llcrypt registered [15538.446665] Key type .llcrypt registered [15538.624540] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [15540.221521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15540.305197] alg: No test for adler32 (adler32-zlib) [15541.665513] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [15542.033092] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [15543.713746] Key type lgssc registered [15545.425382] Lustre: Echo OBD driver; http://www.lustre.org/ [15556.288787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15556.317296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15557.600221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15557.664789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15557.783639] Lustre: lustre-MDT0000: new disk, initializing [15557.905555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15557.967442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15562.716491] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15574.867667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15575.153553] Lustre: 473754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15575.318603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15575.329216] Lustre: Skipped 1 previous similar message [15575.448697] Lustre: lustre-MDT0001: new disk, initializing [15575.671856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15575.747155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15575.772032] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15583.428878] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15596.508435] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15604.694478] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15613.147704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15613.603500] Lustre: lustre-OST0000: new disk, initializing [15613.613544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15613.626980] Lustre: 475724:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15613.767178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15619.206572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15619.223672] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15619.356971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15622.113483] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15633.310921] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15638.235376] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing lnet_if_list [15645.676593] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15645.746751] LNet: Added LNI 192.168.202.118@tcp10 [8/256/0/180] [15653.618083] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15653.650926] LNet: Added LNI 192.168.202.118@tcp11 [8/256/0/180] [15659.904362] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15659.931906] LNet: Added LNI 192.168.202.118@tcp12 [8/256/0/180] [15665.693654] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15665.734554] LNet: Added LNI 192.168.202.118@tcp13 [8/256/0/180] [15671.139496] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15671.198056] LNet: Added LNI 192.168.202.118@tcp14 [8/256/0/180] [15678.118170] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15678.165420] LNet: Added LNI 192.168.202.118@tcp15 [8/256/0/180] [15683.595665] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15683.623579] LNet: Added LNI 192.168.202.118@tcp16 [8/256/0/180] [15689.777983] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15689.807977] LNet: Added LNI 192.168.202.118@tcp17 [8/256/0/180] [15696.746477] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15696.794254] LNet: Added LNI 192.168.202.118@tcp18 [8/256/0/180] [15704.457111] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15704.491285] LNet: Added LNI 192.168.202.118@tcp19 [8/256/0/180] [15708.811609] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15708.837565] LNet: Added LNI 192.168.202.118@tcp20 [8/256/0/180] [15715.660660] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15715.691742] LNet: Added LNI 192.168.202.118@tcp21 [8/256/0/180] [15722.130025] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15722.175418] LNet: Added LNI 192.168.202.118@tcp22 [8/256/0/180] [15727.962959] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15727.994376] LNet: Added LNI 192.168.202.118@tcp23 [8/256/0/180] [15734.310674] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15734.351810] LNet: Added LNI 192.168.202.118@tcp24 [8/256/0/180] [15741.795969] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15741.901027] LNet: Added LNI 192.168.202.118@tcp25 [8/256/0/180] [15748.580792] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15748.618404] LNet: Added LNI 192.168.202.118@tcp26 [8/256/0/180] [15755.858973] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15755.929350] LNet: Added LNI 192.168.202.118@tcp27 [8/256/0/180] [15763.247862] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15763.268493] LNet: Added LNI 192.168.202.118@tcp28 [8/256/0/180] [15770.628442] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15770.647910] LNet: Added LNI 192.168.202.118@tcp29 [8/256/0/180] [15777.819989] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15777.868401] LNet: Added LNI 192.168.202.118@tcp30 [8/256/0/180] [15785.031833] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15785.074826] LNet: Added LNI 192.168.202.118@tcp31 [8/256/0/180] [15792.717621] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15792.859429] LNet: Added LNI 192.168.202.118@tcp32 [8/256/0/180] [15802.068518] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15802.156251] LNet: Added LNI 192.168.202.118@tcp33 [8/256/0/180] [15814.032461] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15814.149881] LNet: Added LNI 192.168.202.118@tcp34 [8/256/0/180] [15823.815499] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15823.862331] LNet: Added LNI 192.168.202.118@tcp35 [8/256/0/180] [15834.510873] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15834.598323] LNet: Added LNI 192.168.202.118@tcp36 [8/256/0/180] [15847.367930] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15847.427945] LNet: Added LNI 192.168.202.118@tcp37 [8/256/0/180] [15857.065270] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15857.108965] LNet: Added LNI 192.168.202.118@tcp38 [8/256/0/180] [15867.029831] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15867.070634] LNet: Added LNI 192.168.202.118@tcp39 [8/256/0/180] [15876.144633] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15876.197619] LNet: Added LNI 192.168.202.118@tcp40 [8/256/0/180] [15884.222651] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15884.264906] LNet: Added LNI 192.168.202.118@tcp41 [8/256/0/180] [15895.390894] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15895.443030] LNet: Added LNI 192.168.202.118@tcp42 [8/256/0/180] [15904.513315] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15904.539355] LNet: Added LNI 192.168.202.118@tcp43 [8/256/0/180] [15913.391479] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15913.427378] LNet: Added LNI 192.168.202.118@tcp44 [8/256/0/180] [15922.646686] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15922.679158] LNet: Added LNI 192.168.202.118@tcp45 [8/256/0/180] [15932.487847] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15932.567973] LNet: Added LNI 192.168.202.118@tcp46 [8/256/0/180] [15942.710304] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15942.747024] LNet: Added LNI 192.168.202.118@tcp47 [8/256/0/180] [15952.406286] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15952.446863] LNet: Added LNI 192.168.202.118@tcp48 [8/256/0/180] [15964.906896] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15964.944554] LNet: Added LNI 192.168.202.118@tcp49 [8/256/0/180] [15974.650223] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15974.692476] LNet: Added LNI 192.168.202.118@tcp50 [8/256/0/180] [15995.361704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15995.376187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15995.389405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15998.435896] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15998.445475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15998.450751] Lustre: Skipped 1 previous similar message [15998.470820] Lustre: Skipped 2 previous similar messages [16001.552655] Lustre: server umount lustre-MDT0000 complete [16006.358245] LustreError: 473746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786763600 with bad export cookie 2166193649415234945 [16006.363207] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16006.382822] LustreError: 473746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16006.861916] Lustre: server umount lustre-MDT0001 complete [16025.057877] Lustre: 473025:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786763603/real 1786763603] req@000000005515b248 x1873556942294784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786763619 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16025.094216] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16028.265407] Lustre: 473025:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786763606/real 1786763606] req@000000009f5aa79b x1873556942294912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786763622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16028.273393] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [16030.184834] Lustre: 473026:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786763608/real 1786763608] req@00000000c8eb0889 x1873556942295040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786763624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16032.288211] Lustre: 473026:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786763610/real 1786763610] req@00000000272d1c3e x1873556942295296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786763626 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16037.344200] Lustre: 473024:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786763615/real 1786763615] req@0000000038f7112c x1873556942295424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786763631 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16038.087044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16051.184475] LustreError: 475717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16051.203937] LustreError: 475717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [16051.266972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16054.309232] LustreError: 475718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16054.314210] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16057.014759] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16059.376372] LustreError: 475717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16064.484721] LustreError: 475718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16066.941080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16067.266551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16070.388364] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16071.904285] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16080.614824] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16088.135875] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16111.591354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16111.621724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16111.630437] Lustre: Skipped 1 previous similar message [16115.001731] Lustre: server umount lustre-OST0000 complete [16121.841368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16121.854957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16121.862956] Lustre: Skipped 2 previous similar messages [16121.875532] Lustre: Skipped 1 previous similar message [16125.805469] Lustre: server umount lustre-MDT0000 complete [16126.946172] LustreError: 499980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16126.976943] LustreError: 499980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16130.020190] LustreError: 473745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786763724 with bad export cookie 2166193649415237654 [16130.024118] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16130.044554] LustreError: 473745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16130.962605] Lustre: server umount lustre-MDT0001 complete [16144.020727] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [16147.854292] Key type lgssc unregistered [16148.237628] LNet: 502782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16148.245904] LNetError: 502782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16148.256560] LNet: Removed LNI 192.168.202.118@tcp [16148.260567] LNet: Removed LNI 192.168.202.118@tcp10 [16148.264763] LNet: Removed LNI 192.168.202.118@tcp11 [16148.269206] LNet: Removed LNI 192.168.202.118@tcp12 [16148.273707] LNet: Removed LNI 192.168.202.118@tcp13 [16148.277905] LNet: Removed LNI 192.168.202.118@tcp14 [16148.281495] LNet: Removed LNI 192.168.202.118@tcp15 [16148.284203] LNet: Removed LNI 192.168.202.118@tcp16 [16148.286695] LNet: Removed LNI 192.168.202.118@tcp17 [16148.289149] LNet: Removed LNI 192.168.202.118@tcp18 [16148.294476] LNet: Removed LNI 192.168.202.118@tcp19 [16148.296300] LNet: Removed LNI 192.168.202.118@tcp20 [16148.301272] LNet: Removed LNI 192.168.202.118@tcp21 [16148.305738] LNet: Removed LNI 192.168.202.118@tcp22 [16148.309751] LNet: Removed LNI 192.168.202.118@tcp23 [16148.314692] LNet: Removed LNI 192.168.202.118@tcp24 [16148.317125] LNet: Removed LNI 192.168.202.118@tcp25 [16148.318788] LNet: Removed LNI 192.168.202.118@tcp26 [16148.322771] LNet: Removed LNI 192.168.202.118@tcp27 [16148.325780] LNet: Removed LNI 192.168.202.118@tcp28 [16148.329907] LNet: Removed LNI 192.168.202.118@tcp29 [16148.332757] LNet: Removed LNI 192.168.202.118@tcp30 [16148.335948] LNet: Removed LNI 192.168.202.118@tcp31 [16148.340790] LNet: Removed LNI 192.168.202.118@tcp32 [16148.344397] LNet: Removed LNI 192.168.202.118@tcp33 [16148.346419] LNet: Removed LNI 192.168.202.118@tcp34 [16148.350549] LNet: Removed LNI 192.168.202.118@tcp35 [16148.353580] LNet: Removed LNI 192.168.202.118@tcp36 [16148.362995] LNet: Removed LNI 192.168.202.118@tcp37 [16148.370780] LNet: Removed LNI 192.168.202.118@tcp38 [16148.377133] LNet: Removed LNI 192.168.202.118@tcp39 [16148.388708] LNet: Removed LNI 192.168.202.118@tcp40 [16148.393603] LNet: Removed LNI 192.168.202.118@tcp41 [16148.397469] LNet: Removed LNI 192.168.202.118@tcp42 [16148.399720] LNet: Removed LNI 192.168.202.118@tcp43 [16148.405870] LNet: Removed LNI 192.168.202.118@tcp44 [16148.407727] LNet: Removed LNI 192.168.202.118@tcp45 [16148.411586] LNet: Removed LNI 192.168.202.118@tcp46 [16148.414232] LNet: Removed LNI 192.168.202.118@tcp47 [16148.417581] LNet: Removed LNI 192.168.202.118@tcp48 [16148.420704] LNet: Removed LNI 192.168.202.118@tcp49 [16148.430145] LNet: Removed LNI 192.168.202.118@tcp50 [16149.354325] Key type .llcrypt unregistered [16149.356532] Key type ._llcrypt unregistered [16172.485335] Key type ._llcrypt registered [16172.487767] Key type .llcrypt registered [16172.614939] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [16186.685707] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [16187.830461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16187.852084] alg: No test for adler32 (adler32-zlib) [16189.041531] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [16189.278899] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [16190.976374] Key type lgssc registered [16192.207039] Lustre: Echo OBD driver; http://www.lustre.org/ [16232.951431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16233.005850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16234.306242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16234.344453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16234.386734] Lustre: lustre-MDT0000: new disk, initializing [16234.440678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16234.450225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16239.517791] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16252.776819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16252.946555] Lustre: 505978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16253.027515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16253.032037] Lustre: Skipped 1 previous similar message [16253.125944] Lustre: lustre-MDT0001: new disk, initializing [16253.231062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16253.274680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16253.291114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16258.408733] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16269.735424] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16276.726995] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16285.125752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16285.438941] Lustre: *** cfs_fail_loc=90f, val=0*** [16285.494659] Lustre: lustre-OST0000: new disk, initializing [16285.496683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16285.499448] Lustre: 508001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16285.573183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16292.529966] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16292.996844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16293.007433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16293.126147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16303.040815] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16313.832585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16313.843043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16313.843637] LustreError: Skipped 1 previous similar message [16313.844281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16313.880903] Lustre: Skipped 1 previous similar message [16317.575452] Lustre: server umount lustre-OST0000 complete [16324.085246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16324.096142] Lustre: Skipped 1 previous similar message [16324.099050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16324.106698] Lustre: Skipped 2 previous similar messages [16329.185824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16329.191355] Lustre: Skipped 1 previous similar message [16334.308217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16334.320067] Lustre: Skipped 1 previous similar message [16336.352178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16336.719863] Lustre: server umount lustre-MDT0000 complete [16340.803015] LustreError: 505972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786763935 with bad export cookie 6691043766659876613 [16340.805314] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16340.817091] LustreError: 505972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16341.235844] Lustre: server umount lustre-MDT0001 complete [16353.623771] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [16356.844916] Key type lgssc unregistered [16357.226522] LNet: 509911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16357.238678] LNetError: 509911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16357.280308] LNet: Removed LNI 192.168.202.118@tcp [16358.339352] Key type .llcrypt unregistered [16358.342109] Key type ._llcrypt unregistered [16381.397463] Key type ._llcrypt registered [16381.399761] Key type .llcrypt registered [16381.486752] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [16382.750392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16382.830070] alg: No test for adler32 (adler32-zlib) [16384.110608] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [16384.454788] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [16386.178295] Key type lgssc registered [16388.025777] Lustre: Echo OBD driver; http://www.lustre.org/ [16395.954861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16395.975276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16397.496973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16402.133485] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16407.022097] LustreError: 511180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16407.037475] LustreError: 511180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16411.208879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16411.493263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16415.884816] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16424.505619] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16430.115596] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16437.311696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16437.626719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16444.428986] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16452.612747] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16468.303451] loop0: detected capacity change from 0 to 204800000 [16468.496547] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16468.502959] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16468.513422] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16475.617298] loop0: detected capacity change from 0 to 204800000 [16475.944844] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16475.970385] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16475.983552] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16483.707578] loop0: detected capacity change from 0 to 204800000 [16491.371277] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16491.557174] Lustre: fs15246-OST0000: new disk, initializing [16491.560547] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16491.565454] Lustre: 514777:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16491.625326] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16499.188532] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16509.444606] loop1: detected capacity change from 0 to 204800000 [16515.543328] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16515.642906] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16515.686388] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16515.751698] Lustre: fs15246-MDT0000: new disk, initializing [16515.855726] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16515.891906] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16516.023317] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16517.098523] Lustre: 515901:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000fc103d2b x1873557825047936/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786764127 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16520.926786] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16526.186375] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16526.199510] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16526.220940] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16544.247467] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16544.296651] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16548.271562] Lustre: server umount fs15246-MDT0000 complete [16549.359673] LustreError: 515741:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16554.468071] LustreError: 512198:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16558.014279] Lustre: server umount fs15246-OST0000 complete [16573.925403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16573.932751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16573.948762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16574.953389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16574.957096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16574.968947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16576.298561] Lustre: server umount lustre-OST0000 complete [16580.075107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16580.092660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16580.098472] Lustre: Skipped 2 previous similar messages [16585.188729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16585.197199] Lustre: Skipped 1 previous similar message [16594.400314] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16594.597688] Lustre: server umount lustre-MDT0000 complete [16595.430860] LustreError: 511180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16595.452250] LustreError: 511180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16599.163449] LustreError: 511160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786764193 with bad export cookie 7939224632521337674 [16599.185689] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16599.194403] LustreError: 511160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16599.624375] Lustre: server umount lustre-MDT0001 complete [16613.281373] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [16618.347515] Key type lgssc unregistered [16618.937620] LNet: 519001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16618.957622] LNetError: 519001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16619.008983] LNet: Removed LNI 192.168.202.118@tcp [16620.360449] Key type .llcrypt unregistered [16620.363185] Key type ._llcrypt unregistered [16643.515444] Key type ._llcrypt registered [16643.518731] Key type .llcrypt registered [16643.655787] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [16644.825035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16644.852444] alg: No test for adler32 (adler32-zlib) [16645.824800] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [16646.048037] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [16647.664189] Key type lgssc registered [16648.703704] Lustre: Echo OBD driver; http://www.lustre.org/ [16695.667677] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [16711.023904] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [16759.565992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16759.618993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16760.963984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16761.003405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16761.135882] Lustre: lustre-MDT0000: new disk, initializing [16761.297289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16761.319670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16766.686956] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16778.617752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16778.778939] Lustre: 523504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16778.848751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16778.855530] Lustre: Skipped 1 previous similar message [16778.911954] Lustre: lustre-MDT0001: new disk, initializing [16779.007813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16779.039376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16779.052303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16784.066429] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16795.020333] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16801.282775] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16809.143834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16809.536691] Lustre: lustre-OST0000: new disk, initializing [16809.540660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16809.548895] Lustre: 525474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16809.648768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16816.060468] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16818.276343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16818.291634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16818.384162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16827.288880] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16832.653500] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16832.891360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16838.225894] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16838.619458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16844.257686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16844.274834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16844.276655] LustreError: Skipped 1 previous similar message [16844.280745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16844.291380] Lustre: Skipped 1 previous similar message [16844.309123] Lustre: Skipped 1 previous similar message [16847.198691] Lustre: server umount lustre-OST0000 complete [16854.498192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16854.507302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16854.518676] Lustre: Skipped 1 previous similar message [16854.528361] Lustre: Skipped 1 previous similar message [16857.813882] Lustre: server umount lustre-MDT0000 complete [16861.728503] LustreError: 523497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786764456 with bad export cookie 578863255007510748 [16861.732983] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16861.735385] LustreError: 523497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16862.150873] Lustre: server umount lustre-MDT0001 complete [16890.080664] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [16901.330380] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [16934.333918] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16603 sec ========= 23:28:47 (1786764527) [16935.875183] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:28:49 (1786764529) === [16938.796450] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:28:52 (1786764532) === [16982.290899] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [16986.669863] Key type lgssc unregistered [16987.184365] LNet: 532124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16987.193691] LNetError: 532124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16987.245617] LNet: Removed LNI 192.168.202.118@tcp [16988.693643] Key type .llcrypt unregistered [16988.695956] Key type ._llcrypt unregistered