[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 526591742 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003233] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008479] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.012040] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014075] *** VALIDATE selinux *** [ 0.022508] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027346] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030129] *** VALIDATE tmpfs *** [ 0.031464] *** VALIDATE proc *** [ 0.032247] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.040725] debug: unmapping init [mem 0xffffffffbd459000-0xffffffffbd460fff] [ 0.042166] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043685] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.045011] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047009] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051341] rcu: Hierarchical SRCU implementation. [ 0.053436] smp: Bringing up secondary CPUs ... [ 0.054584] x86: Booting SMP configuration: [ 0.055025] .... node #0, CPUs: #1 #2 #3 [ 0.059231] smp: Brought up 1 node, 4 CPUs [ 0.061014] smpboot: Max logical packages: 1 [ 0.062014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144703] node 0 deferred pages initialised in 80ms [ 0.147281] devtmpfs: initialized [ 0.148213] x86/mm: Memory block size: 128MB [ 0.150833] gcov: version magic: 0x41383552 [ 0.152271] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154244] pinctrl core: initialized pinctrl subsystem [ 0.155217] [ 0.155764] ************************************************************* [ 0.156012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157011] ** ** [ 0.158013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159012] ** ** [ 0.160012] ** This means that this kernel is built to expose internal ** [ 0.161013] ** IOMMU data structures, which may compromise security on ** [ 0.162012] ** your system. ** [ 0.163012] ** ** [ 0.164011] ** If you see this message and you are not debugging the ** [ 0.165014] ** kernel, report this immediately to your vendor! ** [ 0.166013] ** ** [ 0.167010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168012] ************************************************************* [ 0.169764] NET: Registered protocol family 16 [ 0.170441] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174053] cpuidle: using governor menu [ 0.175604] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177451] PCI: Using configuration type 1 for base access [ 0.179129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189004] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.194018] cryptd: max_cpu_qlen set to 1000 [ 0.196225] ACPI: Added _OSI(Module Device) [ 0.197015] ACPI: Added _OSI(Processor Device) [ 0.199016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.201016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211350] ACPI: Interpreter enabled [ 0.213055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214013] ACPI: Using IOAPIC for interrupt routing [ 0.215109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.218413] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.229130] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.230044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.241318] acpiphp: Slot [2] registered [ 0.243112] acpiphp: Slot [5] registered [ 0.245123] acpiphp: Slot [6] registered [ 0.247168] acpiphp: Slot [7] registered [ 0.249108] acpiphp: Slot [8] registered [ 0.251121] acpiphp: Slot [9] registered [ 0.253181] acpiphp: Slot [10] registered [ 0.255168] acpiphp: Slot [3] registered [ 0.256088] acpiphp: Slot [4] registered [ 0.258101] acpiphp: Slot [11] registered [ 0.260105] acpiphp: Slot [12] registered [ 0.261104] acpiphp: Slot [13] registered [ 0.263228] acpiphp: Slot [14] registered [ 0.264078] acpiphp: Slot [15] registered [ 0.266143] acpiphp: Slot [16] registered [ 0.268100] acpiphp: Slot [17] registered [ 0.270089] acpiphp: Slot [18] registered [ 0.271099] acpiphp: Slot [19] registered [ 0.273111] acpiphp: Slot [20] registered [ 0.275089] acpiphp: Slot [21] registered [ 0.277092] acpiphp: Slot [22] registered [ 0.278122] acpiphp: Slot [23] registered [ 0.280122] acpiphp: Slot [24] registered [ 0.282082] acpiphp: Slot [25] registered [ 0.284114] acpiphp: Slot [26] registered [ 0.286159] acpiphp: Slot [27] registered [ 0.288092] acpiphp: Slot [28] registered [ 0.289108] acpiphp: Slot [29] registered [ 0.291090] acpiphp: Slot [30] registered [ 0.293089] acpiphp: Slot [31] registered [ 0.294080] PCI host bridge to bus 0000:00 [ 0.296018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.299019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.301020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.304021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.307032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.309030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.311246] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.315082] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.318359] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.328759] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.334507] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.337018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.339016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.342020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345026] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.347823] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.350039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.353553] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.361014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.374016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.380017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.389959] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.400015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.412016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.457019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.474049] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.489018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.511017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.566014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.580422] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.591016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.599016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.622017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.632457] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.652018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.674015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.690018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.736993] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.747016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.758017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.777016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.790187] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.804016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.812016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.838018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.852439] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.855351] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.858364] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.860431] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.863207] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.869115] iommu: Default domain type: Passthrough [ 0.871437] SCSI subsystem initialized [ 0.873302] ACPI: bus type USB registered [ 0.874109] usbcore: registered new interface driver usbfs [ 0.876163] usbcore: registered new interface driver hub [ 0.879095] usbcore: registered new device driver usb [ 0.881156] pps_core: LinuxPPS API ver. 1 registered [ 0.883010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.886056] PTP clock support registered [ 0.888068] EDAC MC: Ver: 3.0.0 [ 0.890149] PCI: Using ACPI for IRQ routing [ 0.891844] NetLabel: Initializing [ 0.893010] NetLabel: domain hash size = 128 [ 0.895010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.897092] NetLabel: unlabeled traffic allowed by default [ 0.900044] vgaarb: loaded [ 0.901309] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.903009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.910149] clocksource: Switched to clocksource kvm-clock [ 1.019701] VFS: Disk quotas dquot_6.6.0 [ 1.021427] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.024233] *** VALIDATE ramfs *** [ 1.025649] *** VALIDATE hugetlbfs *** [ 1.027428] pnp: PnP ACPI init [ 1.030112] pnp: PnP ACPI: found 6 devices [ 1.048809] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.052466] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.054965] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.057053] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.060082] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.062967] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.065962] NET: Registered protocol family 2 [ 1.069244] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.075192] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.080140] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.085067] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.087768] TCP: Hash tables configured (established 65536 bind 65536) [ 1.090156] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.093337] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.096104] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.098927] NET: Registered protocol family 1 [ 1.102359] RPC: Registered named UNIX socket transport module. [ 1.104877] RPC: Registered udp transport module. [ 1.106389] RPC: Registered tcp transport module. [ 1.108340] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.111000] NET: Registered protocol family 44 [ 1.125419] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.128270] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.131025] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.133806] PCI: CLS 0 bytes, default 64 [ 1.141468] Unpacking initramfs... [ 2.586321] debug: unmapping init [mem 0xffff8f413cc54000-0xffff8f413ffbffff] [ 2.590377] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.592635] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.595611] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.089595] Initialise system trusted keyrings [ 3.091320] Key type blacklist registered [ 3.093100] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.102863] zbud: loaded [ 3.105939] *** VALIDATE nfs *** [ 3.107404] *** VALIDATE nfs4 *** [ 3.108872] pstore: using deflate compression [ 3.113952] Platform Keyring initialized [ 3.219992] NET: Registered protocol family 38 [ 3.221600] Key type asymmetric registered [ 3.222938] Asymmetric key parser 'x509' registered [ 3.224736] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.227367] io scheduler mq-deadline registered [ 3.228827] io scheduler kyber registered [ 3.230420] io scheduler bfq registered [ 3.232789] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.235371] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.237695] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.240602] ACPI: Power Button [PWRF] [ 3.246027] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.252411] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.270420] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.280262] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.297321] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.322997] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.351670] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.358712] Non-volatile memory driver v1.3 [ 3.361443] Linux agpgart interface v0.103 [ 3.439281] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.444572] vda: detected capacity change from 0 to 74706944 [ 3.473976] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.481591] vdb: detected capacity change from 0 to 1073741824 [ 3.522371] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.529549] vdc: detected capacity change from 0 to 2621440000 [ 3.559592] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.568766] vdd: detected capacity change from 0 to 2621440000 [ 3.603239] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.605887] vde: detected capacity change from 0 to 4294967296 [ 3.631855] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.634266] vdf: detected capacity change from 0 to 4294967296 [ 3.644256] libphy: Fixed MDIO Bus: probed [ 3.675546] usbcore: registered new interface driver usbserial_generic [ 3.677822] usbserial: USB Serial support registered for generic [ 3.680507] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.685675] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.687282] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.690244] mousedev: PS/2 mouse device common for all mice [ 3.697935] rtc_cmos 00:05: RTC can wake from S4 [ 3.701315] rtc_cmos 00:05: registered as rtc0 [ 3.711105] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.713696] intel_pstate: CPU model not supported [ 3.715700] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.726547] hid: raw HID events driver (C) Jiri Kosina [ 3.729204] usbcore: registered new interface driver usbhid [ 3.731544] usbhid: USB HID core driver [ 3.733401] drop_monitor: Initializing network drop monitor service [ 3.736229] Initializing XFRM netlink socket [ 3.738743] NET: Registered protocol family 10 [ 3.738797] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.743503] Segment Routing with IPv6 [ 3.747123] NET: Registered protocol family 17 [ 3.748990] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.749134] mpls_gso: MPLS GSO support [ 3.756675] RAS: Correctable Errors collector initialized. [ 3.758135] AVX version of gcm_enc/dec engaged. [ 3.759179] AES CTR mode by8 optimization enabled [ 3.860736] sched_clock: Marking stable (3860714069, 0)->(4806101105, -945387036) [ 3.865819] registered taskstats version 1 [ 3.868749] Loading compiled-in X.509 certificates [ 3.871825] zswap: loaded using pool lzo/zbud [ 3.921108] Key type big_key registered [ 3.948280] Key type encrypted registered [ 3.952804] ima: No TPM chip found, activating TPM-bypass! [ 3.959224] ima: Allocated hash algorithm: sha1 [ 3.966219] ima: No architecture policies found [ 3.971136] evm: Initialising EVM extended attributes: [ 3.980111] evm: security.selinux [ 3.982010] evm: security.ima [ 3.983686] evm: security.capability [ 3.989702] evm: HMAC attrs: 0x1 [ 3.994360] rtc_cmos 00:05: setting system clock to 2026-08-13 17:17:06 UTC (1786641426) [ 4.018539] debug: unmapping init [mem 0xffffffffbe403000-0xffffffffbe5fffff] [ 4.027580] debug: unmapping init [mem 0xffffffffbd182000-0xffffffffbd458fff] [ 4.042138] Write protecting the kernel read-only data: 28672k [ 4.050214] debug: unmapping init [mem 0xffffffffbb803000-0xffffffffbb9fffff] [ 4.058168] debug: unmapping init [mem 0xffffffffbc114000-0xffffffffbc1fffff] [ 4.229356] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.253570] systemd[1]: Detected virtualization kvm. [ 4.259272] systemd[1]: Detected architecture x86-64. [ 4.267062] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.347235] systemd[1]: No hostname configured. [ 4.354747] systemd[1]: Set hostname to . [ 4.361719] random: systemd: uninitialized urandom read (16 bytes read) [ 4.371066] systemd[1]: Initializing machine ID from random generator. [ 4.570501] random: ln: uninitialized urandom read (6 bytes read) [ 4.937867] random: systemd: uninitialized urandom read (16 bytes read) [ 4.955397] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.985143] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.003387] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.450522] device-mapper: uevent: version 1.0.3 [ 7.466702] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.885071] virtio_net virtio0 ens2: renamed from eth0 [ 9.994218] random: fast init done [ 10.303990] scsi host0: ata_piix [ 10.331465] scsi host1: ata_piix [ 10.340129] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.349028] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.519928] random: crng init done [ 15.524289] random: 7 urandom warning(s) missed due to ratelimiting [ 15.611377] dracut-initqueue[577]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.855480] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.827226] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.846745] SELinux: Disabled at runtime. [ 22.041489] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.054899] systemd[1]: Detected virtualization kvm. [ 22.057977] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.824299] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.834594] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.862966] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.877023] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.894702] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.936671] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.967803] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 24.106831] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 25.579855] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.559723] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.587480] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.251705] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.385484] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 31.720303] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.462367] NFS: Registering the id_resolver key type [ 32.467702] Key type id_resolver registered [ 32.472601] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg246-server login: [ 89.580636] libcfs: loading out-of-tree module taints kernel. [ 89.669879] Key type ._llcrypt registered [ 89.674594] Key type .llcrypt registered [ 89.830323] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 108.917963] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 110.107363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 110.113400] alg: No test for adler32 (adler32-zlib) [ 111.394482] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 112.341115] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 114.055186] Key type lgssc registered [ 115.611980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 119.407549] hrtimer: interrupt took 5467522 ns [ 133.780685] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 178.185170] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 193.115473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 193.141301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.449789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 194.513226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 194.635780] Lustre: lustre-MDT0000: new disk, initializing [ 194.740301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.768275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 200.240828] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.427505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.478838] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 217.499241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 217.502487] Lustre: Skipped 1 previous similar message [ 217.590822] Lustre: lustre-MDT0001: new disk, initializing [ 217.709907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.745598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.755845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 221.339441] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.182983] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 234.791616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.031324] Lustre: lustre-OST0000: new disk, initializing [ 235.037805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 235.043481] Lustre: 8441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.126816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.036740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 238.046519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 238.140505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.618261] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.873117] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 255.983240] Lustre: lustre-OST0001: new disk, initializing [ 255.985805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 255.992872] Lustre: 9514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.079221] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 261.738620] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.265040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 263.282762] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 263.319943] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 272.533592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 279.627580] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 285.785059] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing check_logdir /tmp/testlogs/ [ 290.212424] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing yml_node [ 294.405384] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 296.717911] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 299.216578] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 300.848686] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 13 13:22:01 EDT 2026 [ 316.853289] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 318.275727] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 335.329631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.337858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.349429] Lustre: Skipped 1 previous similar message [ 335.349793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 335.386275] Lustre: Skipped 3 previous similar messages [ 339.860418] Lustre: server umount lustre-MDT0000 complete [ 343.128073] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641765 with bad export cookie 15590056367223543600 [ 343.132599] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.134536] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 345.570229] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.571283] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.571439] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 345.586456] Lustre: Skipped 3 previous similar messages [ 345.613910] LustreError: 6513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 348.575984] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.576924] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.601642] Lustre: Skipped 2 previous similar messages [ 349.703464] Lustre: server umount lustre-MDT0001 complete [ 354.069436] Lustre: server umount lustre-OST0000 complete [ 358.409090] Lustre: server umount lustre-OST0001 complete [ 365.133950] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 375.095329] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 409.379844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.662349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 409.695577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 409.796808] Lustre: lustre-MDT0000: new disk, initializing [ 409.906941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.937283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 415.537530] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 427.415341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.532407] Lustre: 15067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 427.588741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 427.592363] Lustre: Skipped 1 previous similar message [ 427.644526] Lustre: lustre-MDT0001: new disk, initializing [ 427.730496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 427.773471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 427.793537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 432.608172] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 443.005459] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 449.204715] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 456.269967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 456.631654] Lustre: lustre-OST0000: new disk, initializing [ 456.640337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 456.661836] Lustre: 17038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 456.808217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.976853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 457.982605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 458.107623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 462.893903] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 472.674341] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 477.597620] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 477.815907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 482.849921] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 483.061370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 488.943092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.951406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 488.954827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 488.971607] Lustre: Skipped 1 previous similar message [ 489.007284] Lustre: Skipped 1 previous similar message [ 494.070181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 494.078411] Lustre: Skipped 1 previous similar message [ 499.168325] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 499.415116] Lustre: server umount lustre-OST0000 complete [ 504.292037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.294189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 504.303444] Lustre: Skipped 1 previous similar message [ 504.311402] Lustre: Skipped 3 previous similar messages [ 510.591541] Lustre: server umount lustre-MDT0000 complete [ 514.329757] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786641936 with bad export cookie 15590056367223545133 [ 514.331770] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 514.338207] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 514.801618] Lustre: server umount lustre-MDT0001 complete [ 519.780809] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:25:39 (1786641939) [ 522.552766] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 524.394662] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:25:44 (1786641944) [ 531.358454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.135075] LustreError: 19449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.159757] LustreError: 19449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 532.212075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 536.705533] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 537.570495] LustreError: 19450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.680328] LustreError: 19449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.198917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.424855] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 560.551428] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 567.567669] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 577.222365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 577.511310] LustreError: 21269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.545304] LustreError: 21269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 586.443255] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 594.384456] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 603.623650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 603.628228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.631672] LustreError: Skipped 2 previous similar messages [ 603.637986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 603.639439] Lustre: Skipped 1 previous similar message [ 603.668531] Lustre: Skipped 3 previous similar messages [ 606.968748] Lustre: server umount lustre-OST0000 complete [ 613.864836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 613.881137] Lustre: Skipped 1 previous similar message [ 616.796323] Lustre: server umount lustre-MDT0000 complete [ 618.999324] LustreError: 19446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.036291] LustreError: 19446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 620.920211] LustreError: 19431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786642043 with bad export cookie 15590056367223546015 [ 620.933828] LustreError: 19431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 620.934972] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 621.535462] Lustre: server umount lustre-MDT0001 complete [ 634.533356] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 637.514801] Key type lgssc unregistered [ 637.841346] LNet: 22985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.849331] LNetError: 22985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.872841] LNet: Removed LNI 192.168.202.146@tcp [ 638.725514] Key type .llcrypt unregistered [ 638.727869] Key type ._llcrypt unregistered [ 662.828715] Key type ._llcrypt registered [ 662.832029] Key type .llcrypt registered [ 662.938779] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 664.291615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 664.310263] alg: No test for adler32 (adler32-zlib) [ 665.292796] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 665.556800] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 667.199893] Key type lgssc registered [ 668.974911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 677.330466] loop: module loaded [ 677.401034] loop0: detected capacity change from 0 to 204800000 [ 677.580857] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.587492] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 677.971048] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.148414] loop0: detected capacity change from 0 to 204800000 [ 684.381535] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.398320] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.414791] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 689.520310] loop0: detected capacity change from 0 to 204800000 [ 694.506940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 694.530311] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.784068] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 695.859592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 695.970277] Lustre: lustre-MDT0000: new disk, initializing [ 696.109861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.142209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 700.823690] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 709.798191] loop1: detected capacity change from 0 to 204800000 [ 719.284364] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 719.536796] Lustre: lustre-OST07c6: new disk, initializing [ 719.539901] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 719.543922] Lustre: Skipped 1 previous similar message [ 719.548282] Lustre: 26173:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 719.640215] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 727.799820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 727.812792] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 727.857905] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 728.235397] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 733.060817] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 733.070081] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 733.087353] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 733.098950] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 733.113366] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 733.123339] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 749.340996] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 749.355316] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 749.365981] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 749.372269] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 749.381316] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 749.388776] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 754.359113] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 754.368354] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 754.374503] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 754.380185] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.386762] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 754.392131] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.397713] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 754.408904] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.419768] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 754.427560] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.441231] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 754.454708] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.689984] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 757.695581] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 757.700625] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 757.706388] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.713627] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 757.721250] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.734482] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 757.744182] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.751984] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 757.758156] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.763823] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 757.769895] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.782367] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 762.789124] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 762.799284] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 762.811186] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.820370] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 762.827286] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.838075] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 762.850130] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.862401] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 762.874403] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.880833] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 762.887474] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.585811] Lustre: 25162:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 772.596030] Lustre: 25162:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 772.600771] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 772.608542] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.612986] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.616417] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.619534] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 772.625777] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.634490] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 772.645087] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.654090] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.661456] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 825.316924] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 825.320352] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 825.371712] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 827.822154] Lustre: server umount lustre-OST07c6 complete [ 838.304926] Lustre: server umount lustre-MDT0000 complete [ 853.967961] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 856.642924] Key type lgssc unregistered [ 856.916451] LNet: 28736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.926162] LNetError: 28736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.968569] LNet: Removed LNI 192.168.202.146@tcp [ 857.774161] Key type .llcrypt unregistered [ 857.777145] Key type ._llcrypt unregistered [ 876.570902] Key type ._llcrypt registered [ 876.574570] Key type .llcrypt registered [ 876.744730] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 877.780136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 877.919630] alg: No test for adler32 (adler32-zlib) [ 878.910200] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 879.068547] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 880.696993] Key type lgssc registered [ 881.746643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 889.619340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 889.661133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 891.312452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.678334] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 900.585771] LustreError: 30008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.617776] LustreError: 30008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 905.696503] LustreError: 30009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 906.112462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 906.434113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 909.553623] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 915.636212] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 920.431285] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 926.682423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 926.970296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 933.381078] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 936.181449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 941.054953] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 956.901453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 956.902430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.915242] LustreError: Skipped 1 previous similar message [ 956.924227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 956.937680] Lustre: Skipped 1 previous similar message [ 956.961944] Lustre: Skipped 1 previous similar message [ 958.964557] Lustre: server umount lustre-OST0000 complete [ 963.043539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 963.047592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.053186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 967.140376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 967.151916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.560710] Lustre: server umount lustre-MDT0000 complete [ 971.746277] LustreError: 29990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786642394 with bad export cookie 15435936611275137966 [ 971.753354] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 971.755047] LustreError: 29990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 971.972495] Lustre: server umount lustre-MDT0001 complete [ 981.558065] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 984.250120] Key type lgssc unregistered [ 984.498889] LNet: 33745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.509944] LNetError: 33745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.531876] LNet: Removed LNI 192.168.202.146@tcp [ 985.521356] Key type .llcrypt unregistered [ 985.522564] Key type ._llcrypt unregistered [ 1001.037906] Key type ._llcrypt registered [ 1001.042525] Key type .llcrypt registered [ 1001.112636] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1002.091647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1002.180310] alg: No test for adler32 (adler32-zlib) [ 1003.330341] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1003.612754] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1005.319326] Key type lgssc registered [ 1006.627594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1014.689446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1014.716412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.199072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.696459] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1025.511309] LustreError: 34993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1025.534967] LustreError: 34993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1030.662194] LustreError: 34992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1030.815300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.172666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1035.355953] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1041.890190] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1046.586988] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1052.435442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1052.803280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1060.506340] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1060.995555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1068.193989] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1081.833360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1081.835078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.837494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1081.847579] LustreError: Skipped 1 previous similar message [ 1081.867880] Lustre: Skipped 1 previous similar message [ 1086.991850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1086.997854] Lustre: Skipped 2 previous similar messages [ 1087.957752] Lustre: server umount lustre-OST0000 complete [ 1092.577229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1092.595830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.608819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.185462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.190197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.750311] Lustre: server umount lustre-MDT0000 complete [ 1102.304365] LustreError: 35721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.317503] LustreError: 35721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1102.672273] LustreError: 36835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786642525 with bad export cookie 5783644115405743361 [ 1102.679345] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1102.694791] LustreError: 36835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1103.050255] Lustre: server umount lustre-MDT0001 complete [ 1114.644824] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1118.391860] Key type lgssc unregistered [ 1118.769168] LNet: 38554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.778368] LNetError: 38554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.803294] LNet: Removed LNI 192.168.202.146@tcp [ 1119.610067] Key type .llcrypt unregistered [ 1119.612632] Key type ._llcrypt unregistered [ 1142.386419] Key type ._llcrypt registered [ 1142.388215] Key type .llcrypt registered [ 1142.482608] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1143.754136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1143.787654] alg: No test for adler32 (adler32-zlib) [ 1144.812185] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1145.026418] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1146.623161] Key type lgssc registered [ 1147.633993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1155.582435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1155.599643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1157.121937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1161.919283] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1167.336182] LustreError: 39951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.356253] LustreError: 39951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1170.197593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.678460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1175.712776] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1186.153723] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1197.403162] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1205.884842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1206.555664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1212.909351] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1219.660967] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1227.078894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1236.859063] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 1237.474461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1237.482456] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.490182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.528115] Lustre: Skipped 1 previous similar message [ 1241.954153] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 1241.971683] Lustre: Skipped 2 previous similar messages [ 1247.064092] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 1247.078630] Lustre: Skipped 3 previous similar messages [ 1251.296073] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1251.536916] Lustre: server umount lustre-MDT0000 complete [ 1252.182382] LustreError: 39946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.695373] LustreError: 40686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786642677 with bad export cookie 18085873266498198078 [ 1254.696336] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1254.701223] LustreError: 40686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1257.298888] LustreError: 39948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.316145] LustreError: 39948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1257.334217] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping) [ 1257.345610] Lustre: Skipped 3 previous similar messages [ 1257.959961] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.975136] Lustre: Skipped 1 previous similar message [ 1260.015756] LustreError: 39948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.036874] LustreError: 39948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1265.121243] LustreError: 39947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1265.142721] LustreError: 39947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1267.537575] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping) [ 1267.543498] Lustre: Skipped 4 previous similar messages [ 1269.215133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1269.778676] Lustre: server umount lustre-MDT0001 complete [ 1284.261804] Lustre: server umount lustre-OST0000 complete [ 1298.918042] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1301.152238] Key type lgssc unregistered [ 1301.397522] LNet: 43837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.417650] LNetError: 43837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.434977] LNet: Removed LNI 192.168.202.146@tcp [ 1302.205786] Key type .llcrypt unregistered [ 1302.208476] Key type ._llcrypt unregistered [ 1322.737652] Key type ._llcrypt registered [ 1322.740638] Key type .llcrypt registered [ 1322.856327] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1324.226363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1324.267264] alg: No test for adler32 (adler32-zlib) [ 1325.304206] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1325.537706] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1327.216843] Key type lgssc registered [ 1328.162170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1335.487246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1335.502202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1336.878423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.027666] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1347.045040] LustreError: 45243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.056697] LustreError: 45243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1349.837838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.191845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1354.225436] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1360.426976] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1366.231782] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1371.968861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1372.279739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.155431] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1382.510922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1384.529901] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1391.462698] Lustre: lustre-OST0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 1393.141584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.148191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1393.165868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1393.173866] Lustre: Skipped 1 previous similar message [ 1396.584479] Lustre: lustre-OST0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 1397.746982] Lustre: server umount lustre-OST0000 complete [ 1416.672818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1416.675323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.681237] Lustre: Skipped 1 previous similar message [ 1416.683314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1417.711948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1421.825632] Lustre: server umount lustre-MDT0000 complete [ 1422.816618] LustreError: 46280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.836460] LustreError: 46280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1425.428427] LustreError: 48205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786642847 with bad export cookie 6961347803680314344 [ 1425.435113] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1425.443831] LustreError: 48205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1431.980843] Lustre: server umount lustre-MDT0001 complete [ 1443.568316] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1446.804615] Key type lgssc unregistered [ 1447.032382] LNet: 48961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.040206] LNetError: 48961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.102905] LNet: Removed LNI 192.168.202.146@tcp [ 1448.885179] Key type .llcrypt unregistered [ 1448.888113] Key type ._llcrypt unregistered [ 1474.441558] Key type ._llcrypt registered [ 1474.443298] Key type .llcrypt registered [ 1474.632155] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1477.216224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1477.329486] alg: No test for adler32 (adler32-zlib) [ 1478.750723] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1479.306347] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1481.095792] Key type lgssc registered [ 1483.480977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1495.797586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1495.838225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1497.760686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1502.248960] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1508.343682] LustreError: 50379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.378435] LustreError: 50379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1511.373485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1511.775871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1516.505622] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1522.733112] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1527.058939] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1533.246525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1533.572027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.106972] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1542.902376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1548.935403] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1556.059840] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1559.850364] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1571.970812] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1313h43m11s [ 1574.011656] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1577.333681] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 1578.382799] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 1578.464693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1578.472127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.564129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1579.573958] Lustre: Skipped 5 previous similar messages [ 1582.415662] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 1588.571034] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 1588.593700] Lustre: Skipped 3 previous similar messages [ 1590.239449] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1590.445911] Lustre: server umount lustre-MDT0000 complete [ 1591.793815] LustreError: 50379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1591.817924] LustreError: 50379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1594.324649] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1594.713686] LustreError: 51622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.925894] LustreError: 51622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.945886] LustreError: 51622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1602.773639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1602.908364] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1603.176075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.233660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1604.247077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1608.501637] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1612.042494] Lustre: DEBUG MARKER: Wait for df (36708) ... [ 1613.243692] Lustre: DEBUG MARKER: done [ 1618.408178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1618.424961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.443734] Lustre: Skipped 2 previous similar messages [ 1618.469550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1623.271607] Lustre: server umount lustre-OST0000 complete [ 1630.178138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.188858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1630.202026] Lustre: Skipped 2 previous similar messages [ 1630.230062] LustreError: Skipped 1 previous similar message [ 1632.940294] Lustre: server umount lustre-MDT0000 complete [ 1636.790483] LustreError: 50361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643059 with bad export cookie 12120075895368749290 [ 1636.800245] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1636.803623] LustreError: 50361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1637.239586] Lustre: server umount lustre-MDT0001 complete [ 1647.002386] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1649.852395] Key type lgssc unregistered [ 1650.102421] LNet: 55306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1650.107808] LNetError: 55306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1650.123480] LNet: Removed LNI 192.168.202.146@tcp [ 1650.932335] Key type .llcrypt unregistered [ 1650.936184] Key type ._llcrypt unregistered [ 1691.201686] Key type ._llcrypt registered [ 1691.206928] Key type .llcrypt registered [ 1691.358567] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1692.608923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1692.629063] alg: No test for adler32 (adler32-zlib) [ 1693.708115] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1693.929961] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1695.671457] Key type lgssc registered [ 1696.881936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1705.010416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1705.025388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.341639] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1706.354792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1706.467138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1711.043882] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1719.293978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.372758] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1719.395768] Lustre: 57516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1719.570679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1723.803717] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1730.443346] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1736.485229] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1742.883068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1743.042735] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1743.217595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.270393] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1749.668792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1755.949443] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1761.824865] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1313h46m22s [ 1763.123149] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1769.207555] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1771.435583] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1771.489939] LustreError: 57525:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1791.940203] Lustre: lustre-MDT0000: Client 3631894f-7503-4d45-bc55-1ed6fe31924b (at 192.168.202.46@tcp) reconnecting [ 1811.592277] LustreError: 57525:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1815.317495] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1818.057475] Lustre: DEBUG MARKER: done [ 1826.274168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1826.281932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1826.298826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1826.791259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1826.812495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1826.848904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1831.915553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1837.040880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1837.051806] Lustre: Skipped 1 previous similar message [ 1838.047127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1838.152904] Lustre: server umount lustre-OST0000 complete [ 1842.149413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.156593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1842.165432] Lustre: Skipped 1 previous similar message [ 1842.193226] Lustre: Skipped 2 previous similar messages [ 1848.104792] Lustre: server umount lustre-MDT0000 complete [ 1852.080125] LustreError: 57507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643274 with bad export cookie 2567987350908335722 [ 1852.089211] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1852.090996] LustreError: 57507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1852.797610] Lustre: server umount lustre-MDT0001 complete [ 1865.743803] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1870.373366] Key type lgssc unregistered [ 1870.784723] LNet: 61851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.791265] LNetError: 61851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.832130] LNet: Removed LNI 192.168.202.146@tcp [ 1871.880984] Key type .llcrypt unregistered [ 1871.886909] Key type ._llcrypt unregistered [ 1916.273525] Key type ._llcrypt registered [ 1916.275453] Key type .llcrypt registered [ 1916.349524] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1917.353684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1917.403316] alg: No test for adler32 (adler32-zlib) [ 1918.741141] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1919.296532] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1921.071245] Key type lgssc registered [ 1923.176150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1929.137590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1929.174744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.616440] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1930.658794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1930.931835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.206475] Lustre: Failing over lustre-MDT0000 [ 1932.759245] Lustre: server umount lustre-MDT0000 complete [ 1940.604724] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:49:20 (1786643360) [ 1948.776322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.128050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.414476] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1961.111480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.288420] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1961.361935] Lustre: 64648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1961.705203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1966.173571] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1973.238063] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1978.984025] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1985.057516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1985.201577] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1985.353628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.587941] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1994.446270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1997.576729] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2002.601479] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2008.849623] Lustre: server umount lustre-MDT0000 complete [ 2010.084788] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.111413] Lustre: Skipped 1 previous similar message [ 2012.464310] LustreError: 64643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643434 with bad export cookie 3643483707770435228 [ 2012.466852] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2012.474035] LustreError: 64643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2012.827814] Lustre: server umount lustre-MDT0001 complete [ 2016.459741] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2021.825772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2030.495113] Lustre: 63573:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786643437/real 1786643437] req@00000000d0c6e42e x1873431327969408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786643453 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2030.525626] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2030.541842] Lustre: Skipped 1 previous similar message [ 2030.906680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.607172] Lustre: 63575:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786643439/real 1786643439] req@0000000073b1b8e9 x1873431327969536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786643455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2035.680521] Lustre: 63574:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786643442/real 1786643442] req@00000000b272cdb5 x1873431327969664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786643458 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2036.710034] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x32904120d2947f7f [ 2036.750437] Lustre: MGC192.168.202.146@tcp: Connection restored to 0@lo (at 0@lo) [ 2037.278848] LustreError: 66498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.318307] LustreError: 66498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2037.404622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2037.511747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2038.432369] Lustre: 63573:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786643444/real 1786643444] req@00000000a30244bb x1873431327969920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786643460 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2041.505231] LustreError: 66499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.512491] LustreError: 66499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2041.878484] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2042.527664] Lustre: 63572:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786643449/real 1786643449] req@00000000e95ff33b x1873431327970048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786643465 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2046.949812] LustreError: 68419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.969401] LustreError: 68419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2050.019623] LustreError: 68419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.034195] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2050.055050] LustreError: 68419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2050.196025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.467449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2054.330949] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2055.661175] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2060.763222] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2065.370498] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2076.141375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.159365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2076.164137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2078.680784] Lustre: server umount lustre-MDT0000 complete [ 2081.259316] LustreError: 68418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2081.290436] LustreError: 68418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2083.321155] LustreError: 65386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643505 with bad export cookie 3643483707770437503 [ 2083.332271] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2083.345241] LustreError: 65386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2083.913218] Lustre: server umount lustre-MDT0001 complete [ 2092.353979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2101.737829] Lustre: 63572:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786643508/real 1786643508] req@00000000ac4ae57f x1873431327995520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786643524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2101.772806] Lustre: 63572:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2101.776468] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.783043] Lustre: Skipped 2 previous similar messages [ 2105.324866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.679138] LustreError: 71090:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2107.686905] LustreError: 71090:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000003e999ea2 x1873431327995904/t0(0) o250->MGC192.168.202.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786643530 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2107.705716] LustreError: 71090:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2108.145112] LustreError: 69490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2108.265883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.364588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2113.092746] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2121.660406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.343852] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2127.359188] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2131.453768] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2134.864493] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2140.768367] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2151.912442] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.915238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2151.927558] Lustre: Skipped 1 previous similar message [ 2151.946861] Lustre: Skipped 4 previous similar messages [ 2154.415916] Lustre: server umount lustre-MDT0000 complete [ 2157.039627] LustreError: 71104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.057024] LustreError: 71104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2157.809230] LustreError: 64642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643580 with bad export cookie 3643483707770438882 [ 2157.824518] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2157.825245] LustreError: 64642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2158.061437] Lustre: server umount lustre-MDT0001 complete [ 2165.308758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2170.652171] Lustre: DEBUG MARKER: files compared the same [ 2184.116204] Lustre: server umount lustre-OST0000 complete [ 2200.516271] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2203.103484] Key type lgssc unregistered [ 2203.547516] LNet: 74381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.559635] LNetError: 74381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.584945] LNet: Removed LNI 192.168.202.146@tcp [ 2204.295896] Key type .llcrypt unregistered [ 2204.297393] Key type ._llcrypt unregistered [ 2224.171966] Key type ._llcrypt registered [ 2224.176757] Key type .llcrypt registered [ 2224.293717] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2225.442069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2225.531673] alg: No test for adler32 (adler32-zlib) [ 2226.693627] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2226.932072] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2228.608727] Key type lgssc registered [ 2229.780192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2238.156064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2238.169461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.693320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.916742] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug +malloc all [ 2250.215645] LustreError: 75777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.232210] LustreError: 75777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2253.220125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.448618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2258.043327] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug +malloc all [ 2264.971978] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2270.145478] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2276.705200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2277.087593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.459816] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug +malloc all [ 2285.287148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2288.665477] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2298.849081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2298.861391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2298.882963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2300.897597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2300.909654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.910226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2300.942209] Lustre: Skipped 1 previous similar message [ 2303.385869] Lustre: server umount lustre-OST0000 complete [ 2310.111896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2310.117853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.127518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2313.167502] Lustre: server umount lustre-MDT0000 complete [ 2316.255721] LustreError: 75773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.271322] LustreError: 75773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2317.205364] LustreError: 75758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643739 with bad export cookie 6827397191998236828 [ 2317.221756] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2317.223780] LustreError: 75758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2317.730392] Lustre: server umount lustre-MDT0001 complete [ 2330.187463] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2333.546219] Key type lgssc unregistered [ 2333.886956] LNet: 79311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.902082] LNetError: 79311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2333.931354] LNet: Removed LNI 192.168.202.146@tcp [ 2335.491272] Key type .llcrypt unregistered [ 2335.498684] Key type ._llcrypt unregistered [ 2364.764709] Key type ._llcrypt registered [ 2364.768952] Key type .llcrypt registered [ 2364.861843] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2365.698269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2365.870630] alg: No test for adler32 (adler32-zlib) [ 2366.959249] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2367.182400] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2368.831238] Key type lgssc registered [ 2369.862321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2376.819500] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2376.845191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2392.543492] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2397.663427] LustreError: 80701:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.146@tcp: failed processing log, type 4: rc = -110 [ 2424.287209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2429.313635] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2435.172264] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2441.715605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2441.854648] Lustre: *** cfs_fail_loc=706, val=0*** [ 2442.185674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.251167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2446.168168] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2447.329946] LustreError: 80727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.342504] LustreError: 80727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2452.467610] LustreError: 80726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.031493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.348362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2456.977251] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2463.787534] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2468.589923] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2472.929918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2472.942829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2472.970480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2473.952479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2473.959671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.970589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2474.982590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2474.999036] Lustre: Skipped 1 previous similar message [ 2477.089739] Lustre: server umount lustre-OST0000 complete [ 2484.194280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2484.200369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.221457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2486.482205] Lustre: server umount lustre-MDT0000 complete [ 2489.359850] LustreError: 80708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786643911 with bad export cookie 5852708439086999135 [ 2489.369609] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2489.606388] Lustre: server umount lustre-MDT0001 complete [ 2499.742620] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2502.368945] Key type lgssc unregistered [ 2502.650324] LNet: 84319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2502.672471] LNetError: 84319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2502.695461] LNet: Removed LNI 192.168.202.146@tcp [ 2503.634269] Key type .llcrypt unregistered [ 2503.645843] Key type ._llcrypt unregistered [ 2524.351187] Key type ._llcrypt registered [ 2524.353308] Key type .llcrypt registered [ 2524.447060] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2525.281839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2525.292461] alg: No test for adler32 (adler32-zlib) [ 2526.317672] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2526.541478] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2528.191343] Key type lgssc registered [ 2529.235602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2536.025575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2536.050687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2541.808821] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2548.871995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.374899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2553.815733] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2554.848961] LustreError: 86403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.858448] LustreError: 86403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2558.961761] LustreError: 86406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.970504] LustreError: 86406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2562.425625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2562.732949] LustreError: 86403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.755227] LustreError: 86403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2562.789253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.160859] LustreError: 86407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.172579] LustreError: 86407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2568.486854] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2573.289768] LustreError: 86403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.308915] LustreError: 86403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2576.287350] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.301145] Lustre: MGS: Client 1d875341-e9bb-462c-8359-f027d82ecbab (at 0@lo) reconnecting [ 2576.320292] LustreError: 86417:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2576.494732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.556142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2580.302515] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2592.226179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.244985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.254035] Lustre: Skipped 1 previous similar message [ 2597.356073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2597.373757] Lustre: Skipped 1 previous similar message [ 2598.503068] Lustre: server umount lustre-OST0000 complete [ 2607.586499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.618466] Lustre: Skipped 1 previous similar message [ 2607.629946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2607.642616] Lustre: Skipped 1 previous similar message [ 2612.706225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2612.717174] Lustre: Skipped 1 previous similar message [ 2617.311632] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2617.411516] Lustre: server umount lustre-MDT0000 complete [ 2617.830932] LustreError: 86407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.844864] LustreError: 86407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2620.671659] Lustre: server umount lustre-MDT0001 complete [ 2623.823995] Lustre: server umount MGS complete [ 2633.911385] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:00:54 (1786644054) [ 2659.948624] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 2667.873608] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2704.866655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.338606] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2717.573922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2717.753211] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2717.788108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2717.855604] Lustre: lustre-MDT0001: new disk, initializing [ 2717.935268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2717.959520] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2717.970725] Lustre: 93586:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2717.984499] LustreError: 93586:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2722.078399] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2731.332263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2731.568374] Lustre: lustre-OST0000: new disk, initializing [ 2731.571695] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2731.577491] Lustre: 94354:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2731.636876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.147420] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2738.279325] Lustre: 94708:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000007d138d54 x1873431965507072/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786644176 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2745.633296] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2751.446234] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2751.456313] Lustre: MGS: Client 8a1e8275-b671-4396-ba9c-77fb0dbbcd15 (at 0@lo) reconnecting [ 2751.485900] Lustre: 92852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2751.501683] LustreError: 93572:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2751.512949] LustreError: 93572:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2751.521149] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2751.576374] Lustre: lustre-MDT0000: new disk, initializing [ 2751.715437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2751.738644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2751.851048] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2752.801343] Lustre: 95421:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000028347227 x1873431965513600/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786644191 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2755.921489] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2758.633719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2758.653442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2761.900481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2761.910611] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2761.971218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2787.819227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2787.827620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.841513] Lustre: Skipped 1 previous similar message [ 2787.850908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2791.326081] Lustre: server umount lustre-OST0000 complete [ 2799.590384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.594201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2799.615046] Lustre: Skipped 1 previous similar message [ 2799.616066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2799.654452] Lustre: Skipped 3 previous similar messages [ 2801.246466] Lustre: server umount lustre-MDT0000 complete [ 2810.287307] Lustre: server umount lustre-MDT0001 complete [ 2813.865482] Lustre: server umount MGS complete [ 2824.593595] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:04:05 (1786644245) [ 2844.042882] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2846.583638] Key type lgssc unregistered [ 2846.888203] LNet: 97925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2846.898711] LNetError: 97925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2847.918764] LNet: Removed LNI 192.168.202.146@tcp [ 2848.566487] Key type .llcrypt unregistered [ 2848.568808] Key type ._llcrypt unregistered [ 2866.400447] Key type ._llcrypt registered [ 2866.405733] Key type .llcrypt registered [ 2866.500734] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2867.369587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2867.427451] alg: No test for adler32 (adler32-zlib) [ 2868.545962] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2868.822336] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2870.543355] Key type lgssc registered [ 2871.829422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2877.986112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2877.995449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.392245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2884.690499] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2888.673224] LustreError: 99144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.705977] LustreError: 99144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2892.621288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2892.992919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2898.539862] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2906.603095] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2911.247306] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2916.645931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2916.920848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.996979] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2924.099200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2929.980472] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2938.336143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2938.342843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.356654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2939.365920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2939.373693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.375837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2939.400227] Lustre: Skipped 1 previous similar message [ 2939.767212] Lustre: server umount lustre-OST0000 complete [ 2944.484332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.492358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2944.508123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2949.603178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2949.623814] Lustre: Skipped 2 previous similar messages [ 2954.720584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2954.724667] Lustre: Skipped 1 previous similar message [ 2957.279118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2957.458859] Lustre: server umount lustre-MDT0000 complete [ 2959.840287] LustreError: 99145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2959.855078] LustreError: 99145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2960.854674] LustreError: 101926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786644383 with bad export cookie 529928324018764491 [ 2960.858588] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2960.867414] LustreError: 101926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2961.202824] Lustre: server umount lustre-MDT0001 complete [ 2971.885291] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2974.618737] Key type lgssc unregistered [ 2974.954780] LNet: 102682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2974.965204] LNetError: 102682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2974.996904] LNet: Removed LNI 192.168.202.146@tcp [ 2975.756702] Key type .llcrypt unregistered [ 2975.759936] Key type ._llcrypt unregistered [ 2992.472133] Key type ._llcrypt registered [ 2992.475703] Key type .llcrypt registered [ 2992.739213] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2993.985369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2994.112967] alg: No test for adler32 (adler32-zlib) [ 2995.142602] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2995.398726] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2997.071236] Key type lgssc registered [ 2998.249111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3004.311457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3004.316288] LustreError: 103766:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3004.319695] Lustre: Skipped 1 previous similar message [ 3004.319717] LustreError: 103765:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3004.351802] LustreError: 103766:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4972 [ 3004.371676] LustreError: 103766:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3004.382670] LustreError: 103766:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3004.392811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.911765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.470969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.725679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3018.140928] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3022.832689] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing lsmod [ 3025.334405] LustreError: 104967:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3025.345266] LustreError: 104974:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3025.351774] LustreError: 104967:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3025.355278] LustreError: 104967:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3025.360128] LustreError: 104967:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3025.370683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.734682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3029.490382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3029.499567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3029.513143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3031.009149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3031.010549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.035285] Lustre: Skipped 1 previous similar message [ 3036.139958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.155859] Lustre: Skipped 4 previous similar messages [ 3039.202981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3039.221607] Lustre: Skipped 2 previous similar messages [ 3042.273703] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3042.577343] Lustre: server umount lustre-MDT0000 complete [ 3044.328560] LustreError: 103803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3044.343402] LustreError: 103803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3046.931025] LustreError: 103780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786644469 with bad export cookie 7332843973642507070 [ 3046.960467] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3047.415271] LustreError: 104234:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000073fd7763 x1873432456221696/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3047.689819] Lustre: server umount lustre-MDT0001 complete [ 3063.490503] Lustre: server umount lustre-OST0000 complete [ 3073.869965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3074.922437] LustreError: 106039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.011133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.167116] LustreError: 106040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.538485] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3085.285521] LustreError: 106039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3088.873825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.363706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3093.287288] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3101.911873] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3109.723598] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3118.594531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3119.038833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3120.080124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3125.819022] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3134.409907] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3144.674793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3144.675489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.681427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3144.694916] Lustre: Skipped 1 previous similar message [ 3147.701301] Lustre: server umount lustre-OST0000 complete [ 3150.826672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3150.833431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.912142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3154.914422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.918217] Lustre: Skipped 2 previous similar messages [ 3165.151284] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3165.388630] Lustre: server umount lustre-MDT0000 complete [ 3169.175371] LustreError: 106021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786644591 with bad export cookie 7332843973642507777 [ 3169.182796] LustreError: 106021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3169.184046] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3169.570573] Lustre: server umount lustre-MDT0001 complete [ 3182.511784] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 3184.961874] Key type lgssc unregistered [ 3185.150668] LNet: 109578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3185.158655] LNetError: 109578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3185.173836] LNet: Removed LNI 192.168.202.146@tcp [ 3186.019543] Key type .llcrypt unregistered [ 3186.024538] Key type ._llcrypt unregistered [ 3211.789830] Key type ._llcrypt registered [ 3211.792779] Key type .llcrypt registered [ 3211.905206] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3212.899770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3212.951072] alg: No test for adler32 (adler32-zlib) [ 3213.969314] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 3214.158252] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 3215.840191] Key type lgssc registered [ 3217.352734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3227.159916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3227.191115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.797091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3234.304912] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3239.413871] LustreError: 110995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3239.437630] LustreError: 110995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3243.599622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.194722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3249.064751] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3256.342500] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3260.150848] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3265.855748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.262694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3272.957906] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3274.489995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3280.199821] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3285.516554] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3287.848896] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3295.200441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3295.204762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3295.209275] LustreError: Skipped 1 previous similar message [ 3295.211021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3295.225093] Lustre: Skipped 1 previous similar message [ 3295.244341] Lustre: Skipped 1 previous similar message [ 3298.039980] Lustre: server umount lustre-OST0000 complete [ 3305.448129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.458494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3305.462205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3305.466434] Lustre: Skipped 1 previous similar message [ 3308.456691] Lustre: server umount lustre-MDT0000 complete [ 3310.560559] LustreError: 110994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3310.574947] LustreError: 110994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3311.739400] LustreError: 110976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786644734 with bad export cookie 257547909791222997 [ 3311.754868] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3312.029569] Lustre: server umount lustre-MDT0001 complete [ 3327.358488] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 3330.867260] Key type lgssc unregistered [ 3331.347188] LNet: 114636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3331.358690] LNetError: 114636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3332.390650] LNet: Removed LNI 192.168.202.146@tcp [ 3333.409903] Key type .llcrypt unregistered [ 3333.411454] Key type ._llcrypt unregistered [ 3351.551740] Key type ._llcrypt registered [ 3351.553825] Key type .llcrypt registered [ 3351.603507] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3352.200469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3352.234074] alg: No test for adler32 (adler32-zlib) [ 3353.432245] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 3353.735427] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 3355.579753] Key type lgssc registered [ 3356.863632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3367.057710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3367.086112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.828344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.833257] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3379.187815] LustreError: 115870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3379.215244] LustreError: 115870:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3383.855883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.292684] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3384.614500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3390.209395] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3398.339835] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3406.478398] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3415.756943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3416.480540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3422.349200] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3429.340522] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3440.131876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3449.314320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3449.326653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.341111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3450.337386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3450.347342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3450.358742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3450.364552] Lustre: Skipped 1 previous similar message [ 3454.210756] Lustre: server umount lustre-OST0000 complete [ 3460.577139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.582524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3460.598696] Lustre: Skipped 1 previous similar message [ 3460.610054] Lustre: Skipped 1 previous similar message [ 3465.698313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.703388] Lustre: Skipped 1 previous similar message [ 3472.355320] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3472.603577] Lustre: server umount lustre-MDT0000 complete [ 3474.935286] LustreError: 115867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3474.945531] LustreError: 115867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3477.216177] LustreError: 115851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786644899 with bad export cookie 3687249670276609826 [ 3477.224503] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3477.238584] LustreError: 115851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3477.772274] Lustre: server umount lustre-MDT0001 complete [ 3490.925265] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 3493.898697] Key type lgssc unregistered [ 3494.350640] LNet: 119436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3494.369815] LNetError: 119436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3494.385987] LNet: Removed LNI 192.168.202.146@tcp [ 3495.291104] Key type .llcrypt unregistered [ 3495.292872] Key type ._llcrypt unregistered [ 3520.972268] Key type ._llcrypt registered [ 3520.976023] Key type .llcrypt registered [ 3521.147526] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3522.291542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3522.345761] alg: No test for adler32 (adler32-zlib) [ 3523.438392] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 3523.639158] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 3525.426049] Key type lgssc registered [ 3526.570768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3537.525795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3537.548615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3538.984669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3545.015259] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3549.152952] LustreError: 120852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3549.163808] LustreError: 120852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3554.273539] LustreError: 120853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3554.834306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.366980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3559.871325] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3568.243526] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3575.489675] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3583.987885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3584.222161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3585.288057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3591.156867] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3599.384405] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3607.163650] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3612.359157] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3612.364780] Lustre: Skipped 2 previous similar messages [ 3613.896572] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3613.965668] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3617.955607] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3617.965147] Lustre: Skipped 2 previous similar messages [ 3622.988482] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3622.991249] Lustre: Skipped 1 previous similar message [ 3629.480297] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.46@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3629.486726] Lustre: Skipped 2 previous similar messages [ 3638.850460] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.46@tcp 192.168.0.[2,10]@tcp [ 3643.224614] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.46/24@tcp in log lustre-MDT0000 [ 3643.231245] Lustre: Skipped 2 previous similar messages [ 3646.833338] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.46/24@tcp [ 3646.838903] Lustre: Skipped 1 previous similar message [ 3655.141817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3655.148045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.169000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3656.161932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3656.164586] LustreError: Skipped 4 previous similar messages [ 3656.172313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3656.177281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3656.197038] Lustre: Skipped 1 previous similar message [ 3660.678917] Lustre: server umount lustre-OST0000 complete [ 3666.408888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.425503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3666.433530] Lustre: Skipped 1 previous similar message [ 3666.457557] Lustre: Skipped 1 previous similar message [ 3671.523050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3671.546316] Lustre: Skipped 1 previous similar message [ 3675.617231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3675.630457] Lustre: Skipped 1 previous similar message [ 3679.202481] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3679.635036] Lustre: server umount lustre-MDT0000 complete [ 3680.745306] LustreError: 123825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3680.780189] LustreError: 123825:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3683.586313] LustreError: 120833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786645106 with bad export cookie 1930350780122250464 [ 3683.591087] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3683.611392] LustreError: 120833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3684.105516] Lustre: server umount lustre-MDT0001 complete [ 3699.011450] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 3702.319360] Key type lgssc unregistered [ 3702.815790] LNet: 125901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3702.844649] LNetError: 125901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3702.867523] LNet: Removed LNI 192.168.202.146@tcp [ 3703.677907] Key type .llcrypt unregistered [ 3703.679810] Key type ._llcrypt unregistered [ 3739.401556] Key type ._llcrypt registered [ 3739.404218] Key type .llcrypt registered [ 3739.517299] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3741.188746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3741.219133] alg: No test for adler32 (adler32-zlib) [ 3742.500952] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 3742.770841] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 3744.367321] Key type lgssc registered [ 3745.444650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3755.019965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3755.034763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.599633] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3756.645508] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.46/24@tcp [ 3756.676222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3761.769236] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3767.269486] LustreError: 127466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3767.310291] LustreError: 127466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3770.474468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3770.794818] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3770.813672] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.46/24@tcp [ 3770.862277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3775.665582] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3783.049875] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3788.962931] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3798.816083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3799.208465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3802.387495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3807.949670] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3816.856364] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3828.193156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3828.199282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3828.223569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3831.272379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3831.275127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.277113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3834.259785] Lustre: server umount lustre-OST0000 complete [ 3841.505097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3841.507058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3841.525619] Lustre: Skipped 1 previous similar message [ 3841.534162] Lustre: Skipped 1 previous similar message [ 3844.450964] Lustre: server umount lustre-MDT0000 complete [ 3846.624215] LustreError: 127467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.366895] LustreError: 130306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786645270 with bad export cookie 3886180699433321153 [ 3848.371466] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3848.375679] LustreError: 130306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3848.762670] Lustre: server umount lustre-MDT0001 complete [ 3862.875842] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 3866.454282] Key type lgssc unregistered [ 3866.757281] LNet: 131061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3866.765539] LNetError: 131061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3866.794725] LNet: Removed LNI 192.168.202.146@tcp [ 3867.720768] Key type .llcrypt unregistered [ 3867.722936] Key type ._llcrypt unregistered [ 3901.184555] Key type ._llcrypt registered [ 3901.188377] Key type .llcrypt registered [ 3901.266530] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 3919.670679] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3920.978563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3921.045358] alg: No test for adler32 (adler32-zlib) [ 3922.238900] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 3922.646053] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 3924.543514] Key type lgssc registered [ 3925.406258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3968.578382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3968.641922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.126614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3970.185533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3970.318919] Lustre: lustre-MDT0000: new disk, initializing [ 3970.456965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3970.499917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3975.513919] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3987.718825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3987.969410] Lustre: 134466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3988.116199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3988.122523] Lustre: Skipped 1 previous similar message [ 3988.181978] Lustre: lustre-MDT0001: new disk, initializing [ 3988.229983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3988.258234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3988.266124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3993.561368] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4003.289660] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4008.621945] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4015.349135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4015.829888] Lustre: lustre-OST0000: new disk, initializing [ 4015.835593] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4015.845737] Lustre: 136438:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4015.982948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.429638] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4024.375800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4024.385557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4024.467903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4032.499310] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4037.196178] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4037.424089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4041.861843] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4042.079965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4045.288393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.289490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4045.331160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4045.335033] Lustre: Skipped 1 previous similar message [ 4050.264078] Lustre: server umount lustre-OST0000 complete [ 4055.009449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4055.017798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.031133] Lustre: Skipped 1 previous similar message [ 4055.036201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4060.430786] Lustre: server umount lustre-MDT0000 complete [ 4064.084185] LustreError: 134459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786645486 with bad export cookie 10438316651114650881 [ 4064.102433] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4064.649128] Lustre: server umount lustre-MDT0001 complete [ 4075.187706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.900572] LustreError: 138655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4075.914646] LustreError: 138655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4075.976778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4081.123723] LustreError: 138656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4082.159725] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4086.250859] LustreError: 138655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4090.488951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.817277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4096.138175] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4101.863194] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4108.075615] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4115.969430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4116.318307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4122.129333] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4128.665354] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4133.168642] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4133.369849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4138.403388] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4138.660415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4148.504301] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4148.751935] Lustre: lustre-OST0001: new disk, initializing [ 4148.755454] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4148.760680] Lustre: 142027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4148.850729] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4156.102271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4156.117877] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4156.262018] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4156.275735] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4164.374724] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4164.655596] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4169.025580] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4169.249188] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4175.446768] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95fad8009800.ost_server_uuid 50 [ 4176.935996] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95fad8009800.ost_server_uuid in FULL state after 0 sec [ 4186.624449] Lustre: server umount lustre-OST0001 complete [ 4187.112320] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.122543] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4187.123649] Lustre: Skipped 1 previous similar message [ 4187.128914] LustreError: 140476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4187.164493] LustreError: 140476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4192.227292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.231186] LustreError: 140781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.243863] Lustre: Skipped 1 previous similar message [ 4192.246096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4192.261634] LustreError: 140781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4192.280145] Lustre: Skipped 2 previous similar messages [ 4196.567663] Lustre: server umount lustre-OST0000 complete [ 4202.464601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4202.477765] Lustre: Skipped 1 previous similar message [ 4202.480657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4202.492268] Lustre: Skipped 1 previous similar message [ 4206.443762] Lustre: server umount lustre-MDT0000 complete [ 4207.587076] LustreError: 138656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4207.596698] LustreError: 138656:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4209.808101] LustreError: 142025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786645632 with bad export cookie 10438316651114651763 [ 4209.818282] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4209.834553] LustreError: 142025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4210.193131] Lustre: server umount lustre-MDT0001 complete [ 4228.295766] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 4231.007370] Key type lgssc unregistered [ 4231.227837] LNet: 145034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4231.232375] LNetError: 145034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4231.247777] LNet: Removed LNI 192.168.202.146@tcp [ 4231.933184] Key type .llcrypt unregistered [ 4231.936753] Key type ._llcrypt unregistered [ 4281.286657] Key type ._llcrypt registered [ 4281.288295] Key type .llcrypt registered [ 4281.479234] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 4303.584842] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4305.376174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4305.626749] alg: No test for adler32 (adler32-zlib) [ 4306.864200] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 4307.118680] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 4308.879573] Key type lgssc registered [ 4311.066135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4359.042978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4359.084322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4360.384976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4360.414825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4360.522869] Lustre: lustre-MDT0000: new disk, initializing [ 4360.654595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4360.677925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4365.930910] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4378.109509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4378.234396] Lustre: 149197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4378.295987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4378.299993] Lustre: Skipped 1 previous similar message [ 4378.441941] Lustre: lustre-MDT0001: new disk, initializing [ 4378.563480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4378.599333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4378.620739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4382.776810] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4394.738298] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4403.274964] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4411.416157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4411.715349] Lustre: lustre-OST0000: new disk, initializing [ 4411.726245] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4411.732761] Lustre: 151168:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4411.823748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4419.146970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4419.153592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4419.258344] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4419.261330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4428.734471] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4434.066798] Lustre: Failing over lustre-OST0000 [ 4434.314395] Lustre: server umount lustre-OST0000 complete [ 4451.297750] Lustre: 147418:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786645857/real 1786645857] req@00000000a7ffedd1 x1873433831967360/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786645873 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4451.299828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.315432] Lustre: 147418:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4451.326914] Lustre: Skipped 1 previous similar message [ 4452.384643] Lustre: 147418:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786645858/real 1786645858] req@000000000ca3723b x1873433831967872/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786645874 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4452.412668] Lustre: 147418:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4454.027804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4454.427104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4454.446723] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4456.416130] Lustre: 147415:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786645862/real 1786645862] req@000000007a8efb50 x1873433831968640/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786645878 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4456.478375] Lustre: 147415:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4460.596842] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4461.535383] Lustre: 147418:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786645867/real 1786645867] req@0000000032b2e390 x1873433831969792/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786645883 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4462.106113] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4463.019230] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4463.053175] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4466.127178] Lustre: 147418:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786645872/real 1786645872] req@000000008218e3a7 x1873433831970944/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786645888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4466.145996] Lustre: 147418:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4466.877440] Lustre: Failing over lustre-MDT0000 [ 4467.147254] Lustre: server umount lustre-MDT0000 complete [ 4470.789024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4470.809580] Lustre: Skipped 1 previous similar message [ 4473.173810] LustreError: 150476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4473.215400] LustreError: 150476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4475.873358] LustreError: 149202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.902666] LustreError: 149202:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4478.300263] LustreError: 150476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4481.002409] LustreError: 149207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4481.022122] LustreError: 149207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4486.121462] LustreError: 149208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4486.135579] LustreError: 149208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4487.139475] Lustre: 147415:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786645893/real 1786645893] req@00000000cb24148d x1873433831981952/t0(0) o400->MGC192.168.202.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786645909 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4487.184732] Lustre: 147415:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4487.198959] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4487.270566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.681414] LustreError: 149202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4494.716371] LustreError: 149202:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4497.379671] LustreError: 147414:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f8c552c6 x1873433831988224/t0(0) o250->MGC192.168.202.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4497.782267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4497.871809] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4499.802420] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4502.876037] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4503.019340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4503.028040] Lustre: Skipped 1 previous similar message [ 4503.049458] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4503.124898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4513.258187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4513.268287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4513.275165] Lustre: Skipped 2 previous similar messages [ 4517.357813] Lustre: server umount lustre-OST0000 complete [ 4523.498080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4523.502159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4523.522925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4523.538929] Lustre: Skipped 2 previous similar messages [ 4527.383304] Lustre: server umount lustre-MDT0000 complete [ 4528.613412] LustreError: 149204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.646977] LustreError: 149204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4531.472275] LustreError: 151178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786645953 with bad export cookie 2056112711391612495 [ 4531.480822] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4531.484739] LustreError: 151178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4531.853219] Lustre: server umount lustre-MDT0001 complete [ 4543.061791] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 4545.638649] Key type lgssc unregistered [ 4545.963822] LNet: 154831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4545.971844] LNetError: 154831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4545.988598] LNet: Removed LNI 192.168.202.146@tcp [ 4546.811872] Key type .llcrypt unregistered [ 4546.814424] Key type ._llcrypt unregistered [ 4588.688480] Key type ._llcrypt registered [ 4588.696409] Key type .llcrypt registered [ 4588.940195] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4590.749413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4590.875260] alg: No test for adler32 (adler32-zlib) [ 4592.329973] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 4592.843407] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 4594.615259] Key type lgssc registered [ 4596.212354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4606.756512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4606.769883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.250215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4615.560948] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4617.705110] LustreError: 156354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4617.712056] LustreError: 156354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4622.818168] LustreError: 156355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4627.939243] LustreError: 156354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4631.201030] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4633.057209] LustreError: 156355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4635.477950] LustreError: 156383:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4635.504583] LustreError: 156383:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4635.513209] LustreError: 156383:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 28, retries 0, failed: rc = -5 [ 4636.037964] Lustre: server umount lustre-MDT0000 complete [ 4651.236203] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 4663.952464] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4708.964350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4709.334747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4709.382247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4709.544771] Lustre: lustre-MDT0000: new disk, initializing [ 4709.660550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4709.682381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4715.673236] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4728.324419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4728.444592] Lustre: 159690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4728.512403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4728.519351] Lustre: Skipped 1 previous similar message [ 4728.594630] Lustre: lustre-MDT0001: new disk, initializing [ 4728.759859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4728.910265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4728.927540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4734.972255] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4745.968241] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4751.932296] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4757.817829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4757.992467] Lustre: lustre-OST0000: new disk, initializing [ 4757.996373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4758.003557] Lustre: 161660:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4758.099017] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4766.809015] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4766.820912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4766.896245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4767.515739] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4779.748475] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4784.750215] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4787.421483] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:36:46 (1786646206) [ 5038.560524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5038.565876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5038.577689] Lustre: Skipped 1 previous similar message [ 5038.597771] Lustre: Skipped 2 previous similar messages [ 5043.687561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5043.695546] Lustre: Skipped 2 previous similar messages [ 5048.800324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5048.809859] Lustre: Skipped 2 previous similar messages [ 5052.384069] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5052.801111] Lustre: server umount lustre-MDT0000 complete [ 5053.929239] LustreError: 162480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5053.954158] LustreError: 162480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5057.521149] LustreError: 159681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786646480 with bad export cookie 1588164446682259747 [ 5057.523369] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5057.547533] LustreError: 159681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5059.041243] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5059.072378] Lustre: Skipped 1 previous similar message [ 5059.079113] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5062.050271] LustreError: 159697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5062.074816] LustreError: 159697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5064.057176] Lustre: server umount lustre-MDT0001 complete [ 5074.738725] Lustre: server umount lustre-OST0000 complete [ 5085.098835] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 5095.218249] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5135.231060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5135.636738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5135.683812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5135.830657] Lustre: lustre-MDT0000: new disk, initializing [ 5136.036308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5136.080525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5142.954590] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5154.615803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5154.780901] Lustre: 166323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5154.876996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5154.881535] Lustre: Skipped 1 previous similar message [ 5154.932873] Lustre: lustre-MDT0001: new disk, initializing [ 5155.025876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5155.057501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5155.076557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5159.640658] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5168.627525] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5173.998338] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5180.704266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5180.970856] Lustre: lustre-OST0000: new disk, initializing [ 5180.974222] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5180.979594] Lustre: 168291:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5181.074220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5186.987841] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5190.712844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5190.728264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5190.829369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5195.587346] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5200.532556] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5200.798717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5204.643562] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5204.835634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5206.498050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5206.499240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5206.508981] LustreError: Skipped 1 previous similar message [ 5206.512065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5206.514872] Lustre: Skipped 1 previous similar message [ 5206.521873] Lustre: Skipped 2 previous similar messages [ 5212.773561] Lustre: server umount lustre-OST0000 complete [ 5216.739920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5216.745066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5216.749138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5216.766974] Lustre: Skipped 3 previous similar messages [ 5222.332619] Lustre: server umount lustre-MDT0000 complete [ 5225.575944] LustreError: 166317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786646648 with bad export cookie 1588164446682418577 [ 5225.578038] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5225.585087] LustreError: 166317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5225.853800] Lustre: server umount lustre-MDT0001 complete [ 5232.741592] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:44:13 (1786646653) [ 5247.416947] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 5255.946085] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5287.966133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5288.160645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5288.195033] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5288.302197] Lustre: lustre-MDT0000: new disk, initializing [ 5288.386921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5288.439512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5292.910031] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5304.414814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.583346] Lustre: 173045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5305.042184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5311.117214] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5322.175820] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5328.888677] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5335.592478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5335.872166] Lustre: lustre-OST0000: new disk, initializing [ 5335.876100] Lustre: Skipped 1 previous similar message [ 5335.881258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5335.887239] Lustre: Skipped 2 previous similar messages [ 5335.892249] Lustre: 175023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5342.555367] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5343.800658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5343.815834] Lustre: Skipped 1 previous similar message [ 5343.819831] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5343.912385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5353.537472] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5364.707412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5364.735419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5364.758656] Lustre: Skipped 2 previous similar messages [ 5364.766906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5364.766916] Lustre: Skipped 2 previous similar messages [ 5366.716694] Lustre: server umount lustre-OST0000 complete [ 5371.373664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5371.385503] LustreError: Skipped 1 previous similar message [ 5374.948448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5374.973626] Lustre: Skipped 1 previous similar message [ 5377.758570] Lustre: server umount lustre-MDT0000 complete [ 5380.069021] LustreError: 173057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5380.104399] LustreError: 173057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5381.204216] LustreError: 173037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786646803 with bad export cookie 1588164446682419459 [ 5381.215265] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5381.223431] LustreError: 173037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5381.616509] Lustre: server umount lustre-MDT0001 complete [ 5390.784594] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:46:50 (1786646810) [ 5408.093385] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 5418.843683] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5451.922506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5452.166700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5452.190594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5452.248772] Lustre: lustre-MDT0000: new disk, initializing [ 5452.300412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5452.305505] Lustre: Skipped 2 previous similar messages [ 5452.328514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5456.371242] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5465.974121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5466.062810] Lustre: 179345:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5466.269951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5470.092083] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5478.917052] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5483.560090] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5489.377075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5489.625941] Lustre: 181321:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5495.325724] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5496.520462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5503.493552] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5512.168887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5512.170217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5512.176309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5512.176318] Lustre: Skipped 4 previous similar messages [ 5512.186862] Lustre: Skipped 1 previous similar message [ 5512.205142] LustreError: Skipped 1 previous similar message [ 5516.278506] Lustre: server umount lustre-OST0000 complete [ 5522.402693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5525.891807] Lustre: server umount lustre-MDT0000 complete [ 5527.520115] LustreError: 179359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5528.735573] LustreError: 179338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786646951 with bad export cookie 1588164446682420544 [ 5528.738811] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5528.749375] LustreError: 179338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5529.018593] Lustre: server umount lustre-MDT0001 complete [ 5538.547311] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5541.150409] Key type lgssc unregistered [ 5541.531645] LNet: 183181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5541.536706] LNetError: 183181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5541.556681] LNet: Removed LNI 192.168.202.146@tcp [ 5542.518304] Key type .llcrypt unregistered [ 5542.529741] Key type ._llcrypt unregistered [ 5563.908803] Key type ._llcrypt registered [ 5563.911555] Key type .llcrypt registered [ 5564.021498] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5564.821568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5564.845816] alg: No test for adler32 (adler32-zlib) [ 5565.993281] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 5566.252217] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 5567.887235] Key type lgssc registered [ 5568.937076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5575.276476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5575.292863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5576.716641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5580.310457] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5586.916883] LustreError: 184577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5586.929983] LustreError: 184577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5587.072424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5587.437994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5591.788599] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5598.322816] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5602.809586] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5608.697566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5609.034523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.603836] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5621.360934] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5622.398358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5643.232774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5643.233848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5643.256181] LustreError: Skipped 1 previous similar message [ 5643.259437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5643.309798] Lustre: Skipped 1 previous similar message [ 5648.355081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5648.365124] Lustre: Skipped 2 previous similar messages [ 5648.720898] Lustre: server umount lustre-OST0000 complete [ 5658.596636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.600703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5658.618859] Lustre: Skipped 1 previous similar message [ 5662.061793] Lustre: server umount lustre-MDT0000 complete [ 5663.725202] LustreError: 184572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5663.750706] LustreError: 184572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5667.088727] LustreError: 184556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786647089 with bad export cookie 10198549890706985790 [ 5667.110388] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5667.551961] Lustre: server umount lustre-MDT0001 complete [ 5679.189866] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5682.227428] Key type lgssc unregistered [ 5682.479278] LNet: 188133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5682.485507] LNetError: 188133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5682.505366] LNet: Removed LNI 192.168.202.146@tcp [ 5683.540944] Key type .llcrypt unregistered [ 5683.543392] Key type ._llcrypt unregistered [ 5706.806735] Key type ._llcrypt registered [ 5706.809249] Key type .llcrypt registered [ 5706.898667] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5707.779697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5707.857255] alg: No test for adler32 (adler32-zlib) [ 5708.914563] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 5709.106053] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 5710.768845] Key type lgssc registered [ 5711.742362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5719.655492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5719.674950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5721.203981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.626948] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5730.796853] LustreError: 189551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5730.836869] LustreError: 189551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5732.749460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5733.110078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5736.815483] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5746.908879] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5752.280808] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5759.168583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5759.507806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5765.435835] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5765.697102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5773.962949] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5786.085217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5786.101867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5786.128314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5786.994564] Lustre: lustre-OST0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 5788.139061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5788.142834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5788.164245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5790.777742] Lustre: server umount lustre-OST0000 complete [ 5796.868962] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff95fac6421800.ost_server_uuid 50 [ 5798.831606] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5812.711292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.724446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5812.743148] Lustre: Skipped 1 previous similar message [ 5812.756626] Lustre: Skipped 1 previous similar message [ 5817.599776] Lustre: server umount lustre-MDT0000 complete [ 5817.824097] LustreError: 189552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.831175] LustreError: 189552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5820.503918] LustreError: 189532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786647243 with bad export cookie 13254531167627165243 [ 5820.511530] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5820.518514] LustreError: 189532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5820.823554] Lustre: server umount lustre-MDT0001 complete [ 5828.655422] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:54:08 (1786647248) [ 5835.297710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5835.893546] LustreError: 193472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5836.110122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5841.383461] LustreError: 193473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5841.409715] LustreError: 193473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5843.086866] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5846.499248] LustreError: 193472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5853.605129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5854.015495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5858.671214] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5867.642795] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5872.777429] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5879.861777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5880.170470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5882.237137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5887.909840] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5899.697979] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5908.268713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5908.474103] Lustre: lustre-OST0001: new disk, initializing [ 5908.476605] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5908.487041] Lustre: 196315:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5908.598760] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5915.730553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5915.745332] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5915.896434] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5915.995803] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5925.638559] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5929.444715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5929.453912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5929.469229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5929.486401] Lustre: Skipped 1 previous similar message [ 5934.873259] Lustre: server umount lustre-OST0000 complete [ 5936.620378] LustreError: 195294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5936.644561] LustreError: 195294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5941.829379] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5942.080716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5946.293897] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5946.512548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5957.101165] LustreError: 196650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5957.119628] LustreError: 196650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5962.216794] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5962.217756] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5962.236824] LustreError: Skipped 1 previous similar message [ 5962.239831] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5962.239840] Lustre: Skipped 3 previous similar messages [ 5962.295228] Lustre: Skipped 2 previous similar messages [ 5966.940985] Lustre: server umount lustre-OST0001 complete [ 5971.936292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5971.945055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.517742] Lustre: server umount lustre-MDT0000 complete [ 5981.604423] LustreError: 193452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786647404 with bad export cookie 13254531167627166314 [ 5981.637354] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5982.095337] Lustre: server umount lustre-MDT0001 complete [ 6008.814209] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:57:08 (1786647428) [ 6015.403816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6015.615653] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6015.641149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6015.881667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6020.508907] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6029.811908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6029.951043] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6030.040326] Lustre: 199753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6036.558516] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6046.432375] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6053.100785] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6061.592467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6062.035922] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6069.987602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6071.024918] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6078.017498] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6085.203287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6085.285104] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6085.602669] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6085.610364] Lustre: Skipped 2 previous similar messages [ 6091.787798] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6095.237729] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6100.850095] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6105.594749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6105.599477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6105.612761] Lustre: Skipped 2 previous similar messages [ 6105.621459] Lustre: Skipped 5 previous similar messages [ 6118.879151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6119.055373] Lustre: server umount lustre-OST0000 complete [ 6120.936731] LustreError: 201604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6120.961059] LustreError: 201604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6136.300251] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6141.030158] Lustre: server umount lustre-OST0001 complete [ 6146.528683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.539772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6146.551946] Lustre: Skipped 3 previous similar messages [ 6146.560914] Lustre: Skipped 10 previous similar messages [ 6150.487524] Lustre: server umount lustre-MDT0000 complete [ 6154.823244] LustreError: 199745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786647577 with bad export cookie 13254531167627167721 [ 6154.841024] LustreError: 199745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6154.868730] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6155.649135] Lustre: server umount lustre-MDT0001 complete [ 6182.750171] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:00:02 (1786647602) [ 6196.717249] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 6204.895939] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 6239.919611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6240.202322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6240.283904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6240.429794] Lustre: lustre-MDT0000: new disk, initializing [ 6240.538798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.579043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6245.171747] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6255.999969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6256.120311] Lustre: 207865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6256.173750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6256.178173] Lustre: Skipped 1 previous similar message [ 6256.274591] Lustre: lustre-MDT0001: new disk, initializing [ 6256.446041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6256.471278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6261.203466] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6271.977856] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6279.598420] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6289.617505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6290.128231] Lustre: lustre-OST0000: new disk, initializing [ 6290.134469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6290.161574] Lustre: 209832:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6297.694485] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6299.694024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6299.703044] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6299.805159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6306.327112] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6311.078059] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6311.306482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6317.730636] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6318.473259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6320.609480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6320.620875] LustreError: Skipped 1 previous similar message [ 6320.622055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.639499] Lustre: Skipped 1 previous similar message [ 6320.647098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6320.660369] Lustre: Skipped 1 previous similar message [ 6326.949368] Lustre: server umount lustre-OST0000 complete [ 6339.866539] Lustre: server umount lustre-MDT0000 complete [ 6341.107025] LustreError: 207874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.138636] LustreError: 207874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6347.227874] LustreError: 207857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786647769 with bad export cookie 13254531167627169513 [ 6347.242265] LustreError: 207857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6347.251404] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6347.754710] Lustre: server umount lustre-MDT0001 complete [ 6359.507685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6364.353165] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6373.212241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6379.069985] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6387.829581] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6393.809389] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6400.912874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6409.558891] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6418.495310] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6423.345593] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6423.560362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6428.617813] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6428.927801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6434.310885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6434.316914] LustreError: Skipped 1 previous similar message [ 6437.334304] Lustre: server umount lustre-OST0000 complete [ 6446.311202] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6450.238156] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6479.857648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6488.425709] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6498.137640] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6498.668614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.678686] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.681815] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6498.722138] Lustre: Skipped 1 previous similar message [ 6500.307724] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6509.055607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.063893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6509.105728] Lustre: Skipped 5 previous similar messages [ 6509.106315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6509.142587] Lustre: Skipped 8 previous similar messages [ 6510.575442] Lustre: server umount lustre-OST0000 complete [ 6523.147226] Lustre: server umount lustre-MDT0000 complete [ 6524.385452] LustreError: 212779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6524.398131] LustreError: 212779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6527.588786] LustreError: 212031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786647950 with bad export cookie 13254531167627170395 [ 6527.609082] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6527.622557] LustreError: 212031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6528.008262] Lustre: server umount lustre-MDT0001 complete [ 6536.575426] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:05:56 (1786647956) [ 6543.262925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6543.889908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6543.893105] Lustre: Skipped 6 previous similar messages [ 6548.063420] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6556.163733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6561.516459] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6569.151823] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6575.127862] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6582.123612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6588.897469] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6595.984381] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6601.042411] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6601.317178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6605.895760] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6605.986350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6611.935088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6612.131332] Lustre: lustre-OST0001: new disk, initializing [ 6612.138799] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6612.144718] Lustre: 221394:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6618.773162] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6619.247135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6619.254470] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6619.337240] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6627.678399] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6632.809862] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6633.165346] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6638.474664] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6638.740797] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6645.216871] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6645.234816] LustreError: Skipped 3 previous similar messages [ 6646.888371] Lustre: server umount lustre-OST0001 complete [ 6653.963529] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6654.131433] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6658.248778] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6658.408872] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6687.470207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6692.750858] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6692.860618] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6692.882407] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6692.883292] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6692.902807] Lustre: Skipped 1 previous similar message [ 6699.772499] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6702.033523] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6710.679584] Lustre: server umount lustre-OST0001 complete [ 6733.839992] Lustre: server umount lustre-OST0000 complete [ 6745.743335] Lustre: server umount lustre-MDT0000 complete [ 6750.319672] LustreError: 218003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648172 with bad export cookie 13254531167627171361 [ 6750.325284] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6750.334797] LustreError: 218003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6757.124930] Lustre: server umount lustre-MDT0001 complete [ 6783.988072] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:10:04 (1786648204) [ 6791.307025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6791.473410] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6791.518483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6797.690370] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6807.658539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6807.787918] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6807.818483] Lustre: 226639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6812.333167] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6818.953550] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6825.421924] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6833.401174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6833.585534] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6840.525521] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6849.362246] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6858.727303] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6858.811783] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6865.337947] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6873.293655] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6874.976664] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6879.886038] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6880.110184] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6884.413803] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6884.761560] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6890.755049] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95fac7c95000.ost_server_uuid 50 [ 6893.489728] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95fac7c95000.ost_server_uuid in IDLE state after 0 sec [ 6894.797656] Lustre: Permanently deactivating lustre-OST0001 [ 6894.808687] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6900.220138] Lustre: Permanently reactivating lustre-OST0001 [ 6900.231121] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6900.235277] Lustre: Skipped 2 previous similar messages [ 6905.828461] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6905.839453] LustreError: Skipped 2 previous similar messages [ 6905.846931] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6905.889364] Lustre: Skipped 10 previous similar messages [ 6905.905299] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6905.919508] Lustre: Skipped 12 previous similar messages [ 6911.566819] Lustre: server umount lustre-OST0001 complete [ 6914.037817] LustreError: 228902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6914.072692] LustreError: 228902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 43 previous similar messages [ 6930.399331] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6930.754274] Lustre: server umount lustre-OST0000 complete [ 6942.550088] Lustre: server umount lustre-MDT0000 complete [ 6947.683034] LustreError: 226629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648370 with bad export cookie 13254531167627172649 [ 6947.688249] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6947.704240] LustreError: 226629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6948.052775] Lustre: server umount lustre-MDT0001 complete [ 6970.995973] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:13:11 (1786648391) [ 6978.875348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6979.417691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6979.469184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6979.483633] Lustre: Skipped 2 previous similar messages [ 6985.198352] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 6992.986734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6993.063330] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6993.078117] Lustre: 233333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6999.486471] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7009.333984] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7016.321457] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7023.763451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7023.961521] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7023.987587] Lustre: Permanently deactivating lustre-OST0000 [ 7024.002216] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7032.384676] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7032.596158] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7040.707757] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7047.335565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7047.466798] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7054.595108] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7056.513953] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7065.914788] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7067.456023] Lustre: Permanently reactivating lustre-OST0000 [ 7067.468756] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7067.473179] Lustre: Skipped 2 previous similar messages [ 7075.365076] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7075.368820] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 43 seconds. I think it's dead, and I am evicting it. exp 00000000bbe49099, cur 1786648497 deadline 1786648484 last 1786648454 [ 7075.379138] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7075.389628] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7075.473471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7083.105852] Lustre: server umount lustre-OST0001 complete [ 7093.469378] Lustre: server umount lustre-OST0000 complete [ 7104.885512] Lustre: server umount lustre-MDT0000 complete [ 7109.948603] LustreError: 233325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648532 with bad export cookie 13254531167627174308 [ 7109.952153] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7109.958515] LustreError: 233325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7110.357618] Lustre: server umount lustre-MDT0001 complete [ 7124.617119] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 7126.786832] Key type lgssc unregistered [ 7127.050561] LNet: 238217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7127.055280] LNetError: 238217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7127.070980] LNet: Removed LNI 192.168.202.146@tcp [ 7127.766451] Key type .llcrypt unregistered [ 7127.768554] Key type ._llcrypt unregistered [ 7152.384811] Key type ._llcrypt registered [ 7152.387797] Key type .llcrypt registered [ 7152.477984] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7153.668571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7153.688732] alg: No test for adler32 (adler32-zlib) [ 7154.716870] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 7155.030750] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 7156.689104] Key type lgssc registered [ 7157.786403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7168.490237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7168.505940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7170.060634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7174.794628] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7180.273349] LustreError: 239686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7180.301966] LustreError: 239686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7184.030959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7184.165933] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7184.219526] Lustre: 239676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7184.229423] Lustre: Permanently deactivating lustre-MDT0001 [ 7184.243943] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7184.303900] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7184.318216] LustreError: 239716:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7184.329365] LustreError: 239716:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7184.337531] LustreError: 239716:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 7184.601236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7189.569945] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7197.893074] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7203.531184] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7210.724918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7211.193439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7215.277476] LustreError: 241526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7215.291258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7220.180366] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7220.711612] LustreError: 241525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7225.833486] LustreError: 241720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7229.955542] LustreError: 241527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7229.977936] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7229.983682] LustreError: 241527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7230.878189] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7239.762023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7240.078298] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7245.332012] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7245.347359] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7247.711836] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7258.480908] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7262.904924] Lustre: Permanently reactivating lustre-MDT0001 [ 7262.918357] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7262.922513] Lustre: Skipped 2 previous similar messages [ 7266.702625] Lustre: lustre-MDT0001: Client 3e6a1ebd-4967-4d70-aa64-29b85f985a02 (at 192.168.202.46@tcp) reconnecting [ 7275.462739] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7277.742472] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7284.481349] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7284.711765] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7290.941655] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7291.261240] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7295.142753] Lustre: Permanently deactivating lustre-MDT0001 [ 7295.161874] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7295.171717] Lustre: Skipped 2 previous similar messages [ 7303.413432] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7311.863364] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.870108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.874380] Lustre: Skipped 2 previous similar messages [ 7311.914966] Lustre: Skipped 3 previous similar messages [ 7316.962285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7316.974920] Lustre: Skipped 3 previous similar messages [ 7322.084751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7322.090494] Lustre: Skipped 1 previous similar message [ 7326.175155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7326.469826] Lustre: server umount lustre-MDT0000 complete [ 7327.216795] LustreError: 239686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7327.239888] LustreError: 239686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7332.277845] LustreError: 239667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648754 with bad export cookie 7653433246781526743 [ 7332.283643] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7332.290854] LustreError: 239667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7332.887957] Lustre: server umount lustre-MDT0001 complete [ 7338.288484] Lustre: server umount lustre-OST0000 complete [ 7352.180731] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:19:32 (1786648772) [ 7367.052494] Lustre: server umount lustre-OST0001 complete [ 7376.222497] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 7389.762383] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7433.171507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7433.433668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7433.440191] Lustre: Skipped 2 previous similar messages [ 7433.468719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7433.539788] Lustre: lustre-MDT0000: new disk, initializing [ 7433.608469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7433.625458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7438.621829] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7451.014780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7451.258122] Lustre: 247952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7451.341288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7451.347939] Lustre: Skipped 1 previous similar message [ 7451.498258] Lustre: lustre-MDT0001: new disk, initializing [ 7451.643884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7451.717016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7451.735514] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7456.937819] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7468.577058] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7475.444408] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7483.357028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7483.779623] Lustre: lustre-OST0000: new disk, initializing [ 7483.787912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7483.803159] Lustre: 249922:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7483.903788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7489.583716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7489.597690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7489.696986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7494.133264] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7506.171449] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7508.350548] LustreError: 247961:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7517.711839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7517.893658] Lustre: lustre-OST0001: new disk, initializing [ 7517.904460] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7517.918313] Lustre: 251037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7518.046667] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7518.440303] LustreError: 247961:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7518.476651] LustreError: 249030:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7524.653741] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7526.565142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7526.586339] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7526.732247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7528.519072] LustreError: 249030:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7535.073713] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7537.206454] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7542.245381] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7542.248458] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7542.256775] Lustre: Skipped 1 previous similar message [ 7542.259660] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7545.591932] Lustre: server umount lustre-OST0001 complete [ 7547.374594] LustreError: 251482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.416750] LustreError: 251482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7566.815941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7566.817437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7566.824064] LustreError: Skipped 1 previous similar message [ 7566.832160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7566.842783] Lustre: Skipped 2 previous similar messages [ 7566.873966] Lustre: Skipped 2 previous similar messages [ 7569.609268] Lustre: server umount lustre-OST0000 complete [ 7574.496693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7574.502295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.083733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7578.086937] Lustre: Skipped 3 previous similar messages [ 7580.160324] Lustre: server umount lustre-MDT0000 complete [ 7583.207972] LustreError: 249030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7583.235526] LustreError: 249030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7584.035437] LustreError: 247944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786649006 with bad export cookie 7653433246781529557 [ 7584.045553] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7584.056052] LustreError: 247944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7590.577965] Lustre: server umount lustre-MDT0001 complete [ 7603.772808] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 7607.487731] Key type lgssc unregistered [ 7607.819439] LNet: 253076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7607.844369] LNetError: 253076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7607.858872] LNet: Removed LNI 192.168.202.146@tcp [ 7608.558430] Key type .llcrypt unregistered [ 7608.560262] Key type ._llcrypt unregistered [ 7656.736855] Key type ._llcrypt registered [ 7656.742514] Key type .llcrypt registered [ 7656.947433] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7657.970522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7658.246529] alg: No test for adler32 (adler32-zlib) [ 7659.622995] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 7660.034111] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 7661.895502] Key type lgssc registered [ 7664.358420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7674.423948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7674.437443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.762500] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7675.789687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7675.930418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7679.812506] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7690.723517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7690.803064] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7690.831076] Lustre: 255285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7691.040644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7696.160427] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7705.080554] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7714.178825] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7720.992412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7721.490577] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7721.712254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7727.842518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7728.402832] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7738.878520] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7794.144609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7794.149713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7794.156796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7794.659637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7794.683182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7798.838644] Lustre: server umount lustre-OST0000 complete [ 7804.493185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7819.276585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7819.632225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7826.617180] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7834.983179] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7835.013467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7835.014783] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7835.045276] Lustre: Skipped 1 previous similar message [ 7836.644943] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7881.189099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7881.193245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7881.205949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7881.214387] Lustre: Skipped 1 previous similar message [ 7884.881107] Lustre: server umount lustre-OST0000 complete [ 7890.923048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7890.930671] LustreError: Skipped 1 previous similar message [ 7890.934722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7890.950557] Lustre: Skipped 1 previous similar message [ 7890.959384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7896.441881] Lustre: server umount lustre-MDT0000 complete [ 7901.652152] LustreError: 255279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786649324 with bad export cookie 563757168192712718 [ 7901.656543] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7901.663863] LustreError: 255279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7902.117421] Lustre: server umount lustre-MDT0001 complete [ 7915.267286] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 7918.818165] Key type lgssc unregistered [ 7919.111866] LNet: 260713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7919.117709] LNetError: 260713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7919.129514] LNet: Removed LNI 192.168.202.146@tcp [ 7920.248302] Key type .llcrypt unregistered [ 7920.250278] Key type ._llcrypt unregistered [ 7949.051954] Key type ._llcrypt registered [ 7949.058865] Key type .llcrypt registered [ 7949.180754] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 7950.199879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7950.250763] alg: No test for adler32 (adler32-zlib) [ 7951.468849] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 7951.756412] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 7953.471191] Key type lgssc registered [ 7954.875459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7964.492433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7964.524449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7966.161649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7971.838979] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7976.424495] LustreError: 262132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7976.453216] LustreError: 262132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7981.539167] LustreError: 262131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7982.477585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7982.978422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7988.492912] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 7997.537324] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8005.751605] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8013.880986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8014.453458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8021.977759] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8031.776343] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8035.979378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8063.968755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8063.981406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8064.010230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8067.044101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8067.044561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8067.081975] Lustre: Skipped 1 previous similar message [ 8068.362502] Lustre: server umount lustre-OST0000 complete [ 8075.238807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8075.249368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8075.280934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8077.280047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8077.313524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8079.495338] Lustre: server umount lustre-MDT0000 complete [ 8082.404305] LustreError: 263080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8082.424353] LustreError: 263080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8083.998060] LustreError: 262111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786649506 with bad export cookie 9510335474568031622 [ 8084.016584] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8084.366539] Lustre: server umount lustre-MDT0001 complete [ 8099.100730] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8102.570827] Key type lgssc unregistered [ 8102.823751] LNet: 266444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8102.850250] LNetError: 266444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8103.916168] LNet: Removed LNI 192.168.202.146@tcp [ 8105.166462] Key type .llcrypt unregistered [ 8105.170429] Key type ._llcrypt unregistered [ 8125.016611] Key type ._llcrypt registered [ 8125.022740] Key type .llcrypt registered [ 8125.170515] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8126.607384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8126.639133] alg: No test for adler32 (adler32-zlib) [ 8128.051912] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 8128.379799] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 8130.127171] Key type lgssc registered [ 8131.808985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8141.888214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8141.950869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8143.658713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8148.815273] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8154.095516] LustreError: 267667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8154.113875] LustreError: 267667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8158.264900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8158.810346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8164.576552] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8173.365481] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8179.628979] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8187.193198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8187.577743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8188.608500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8194.911431] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8204.127523] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8217.746372] Lustre: server umount lustre-OST0000 complete [ 8224.738160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8224.743275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8224.769846] Lustre: Skipped 1 previous similar message [ 8229.857207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8229.864883] Lustre: Skipped 1 previous similar message [ 8232.155767] Lustre: server umount lustre-MDT0000 complete [ 8235.000036] LustreError: 267668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8235.028498] LustreError: 267668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8236.003920] Lustre: 266980:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786649642/real 1786649642] req@0000000016699bba x1873437838569344/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786649658 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8236.035166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8236.056272] LustreError: 270003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8236.068313] LustreError: 270003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8236.098314] LustreError: 267648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786649658 with bad export cookie 5978086984704159112 [ 8236.107650] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8236.396959] Lustre: server umount lustre-MDT0001 complete [ 8248.552894] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8251.896296] Key type lgssc unregistered [ 8252.216676] LNet: 271349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8252.226305] LNetError: 271349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8252.248469] LNet: Removed LNI 192.168.202.146@tcp [ 8253.326098] Key type .llcrypt unregistered [ 8253.327815] Key type ._llcrypt unregistered [ 8283.652477] Key type ._llcrypt registered [ 8283.656749] Key type .llcrypt registered [ 8283.863313] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8285.695504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8285.809298] alg: No test for adler32 (adler32-zlib) [ 8287.153897] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 8287.500527] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 8289.202157] Key type lgssc registered [ 8290.887592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8301.267672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8301.319280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8303.330165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8309.805508] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8312.803195] LustreError: 272767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8312.819521] LustreError: 272767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8317.924432] LustreError: 272768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8320.312756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8320.838933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8326.316847] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8335.997353] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8342.756023] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8350.323808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8350.660635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8359.077440] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8368.370747] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8373.186892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8401.909862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8401.926564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8401.962892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8403.938727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8403.956906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8403.961837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8404.007882] Lustre: Skipped 1 previous similar message [ 8406.352491] Lustre: server umount lustre-OST0000 complete [ 8413.154976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8413.176693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8413.194456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8417.979546] Lustre: server umount lustre-MDT0000 complete [ 8419.311533] LustreError: 272768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8419.331598] LustreError: 272768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8422.760360] LustreError: 272749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786649845 with bad export cookie 5113048962900904738 [ 8422.771494] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8422.773862] LustreError: 272749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8423.311408] Lustre: server umount lustre-MDT0001 complete [ 8437.717623] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8442.095472] Key type lgssc unregistered [ 8442.498820] LNet: 277079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8442.509094] LNetError: 277079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8442.531332] LNet: Removed LNI 192.168.202.146@tcp [ 8443.897638] Key type .llcrypt unregistered [ 8443.899473] Key type ._llcrypt unregistered [ 8465.656860] Key type ._llcrypt registered [ 8465.659765] Key type .llcrypt registered [ 8465.834330] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8467.706874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8467.750842] alg: No test for adler32 (adler32-zlib) [ 8468.953507] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 8469.202374] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 8470.919333] Key type lgssc registered [ 8472.540690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8483.158655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8483.175516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8484.815535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8490.674785] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8495.077718] LustreError: 278323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8495.097284] LustreError: 278323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8500.192883] LustreError: 278325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8501.904097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8502.629643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8508.937639] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8517.085744] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8523.837974] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8532.875288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8533.539853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8541.089537] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8549.389890] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8555.137878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8567.265974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8567.275591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8567.300769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8568.198577] Lustre: server umount lustre-OST0000 complete [ 8574.432902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8574.449436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8574.468737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8575.974317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8575.982814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8581.088691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8581.097832] Lustre: Skipped 2 previous similar messages [ 8583.265431] Lustre: server umount lustre-MDT0000 complete [ 8586.209678] LustreError: 278336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8586.261302] LustreError: 278336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8587.324240] LustreError: 278290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650009 with bad export cookie 9547142882613484780 [ 8587.333741] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8587.350893] LustreError: 278290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8587.931394] Lustre: server umount lustre-MDT0001 complete [ 8600.537881] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8603.507761] Key type lgssc unregistered [ 8603.872763] LNet: 282028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8603.877842] LNetError: 282028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8603.893807] LNet: Removed LNI 192.168.202.146@tcp [ 8604.786911] Key type .llcrypt unregistered [ 8604.789360] Key type ._llcrypt unregistered [ 8617.317839] Key type ._llcrypt registered [ 8617.319845] Key type .llcrypt registered [ 8617.441435] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8635.939794] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8653.913401] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8655.296171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8655.484449] alg: No test for adler32 (adler32-zlib) [ 8656.792330] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 8657.206946] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 8658.959202] Key type lgssc registered [ 8660.117783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8698.958541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8699.036909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8700.516979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8700.559692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8700.618628] Lustre: lustre-MDT0000: new disk, initializing [ 8700.763355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8700.789838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8706.996929] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8719.029680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8719.191484] Lustre: 285581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8719.265479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8719.269820] Lustre: Skipped 1 previous similar message [ 8719.364263] Lustre: lustre-MDT0001: new disk, initializing [ 8719.501720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8719.541238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8719.551562] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8724.818450] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8734.355784] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8740.204367] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8747.153393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8747.438161] Lustre: lustre-OST0000: new disk, initializing [ 8747.441689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8747.453262] Lustre: 287554:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8747.545883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8753.058519] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8753.691565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8753.702351] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8753.887799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8762.532421] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8767.676824] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8767.937472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8773.474929] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8773.703270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8779.233602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8779.248418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8779.272163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8779.755789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8779.756843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8779.779236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8782.006478] Lustre: server umount lustre-OST0000 complete [ 8789.988084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8789.992470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8789.995425] Lustre: Skipped 1 previous similar message [ 8790.013523] Lustre: Skipped 1 previous similar message [ 8792.537437] Lustre: server umount lustre-MDT0000 complete [ 8796.454212] LustreError: 285575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650218 with bad export cookie 18351398054957818180 [ 8796.474863] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8797.079620] Lustre: server umount lustre-MDT0001 complete [ 8806.595631] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:43:46 (1786650226) [ 8814.348786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8814.953113] LustreError: 289966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8814.975231] LustreError: 289966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8815.089378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8820.194761] LustreError: 289967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8820.204528] LustreError: 289967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8820.751089] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8825.334720] LustreError: 289966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8830.433843] LustreError: 289967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8830.766294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8831.176824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8836.274988] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8845.636277] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8851.317961] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8858.193287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8858.775776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8865.536517] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 8873.702683] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8885.730109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8885.742377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8885.760129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8890.677906] Lustre: server umount lustre-OST0000 complete [ 8897.000696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8897.012812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8897.013074] Lustre: Skipped 2 previous similar messages [ 8897.033333] Lustre: Skipped 3 previous similar messages [ 8901.128393] Lustre: server umount lustre-MDT0000 complete [ 8902.116264] LustreError: 290965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8902.145601] LustreError: 290965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8906.046813] LustreError: 289948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650328 with bad export cookie 18351398054957819062 [ 8906.077857] LustreError: 289948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8906.078445] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8906.373338] Lustre: server umount lustre-MDT0001 complete [ 8919.231488] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 8922.634897] Key type lgssc unregistered [ 8923.088195] LNet: 293504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8923.103828] LNetError: 293504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8923.130803] LNet: Removed LNI 192.168.202.146@tcp [ 8924.327968] Key type .llcrypt unregistered [ 8924.330131] Key type ._llcrypt unregistered [ 8956.750353] Key type ._llcrypt registered [ 8956.755559] Key type .llcrypt registered [ 8956.990422] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 8974.039222] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 8975.578343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8975.930509] alg: No test for adler32 (adler32-zlib) [ 8977.229994] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 8977.597957] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 8979.376263] Key type lgssc registered [ 8980.835270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9033.313406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9033.388857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9034.685943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9034.727378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9034.842419] Lustre: lustre-MDT0000: new disk, initializing [ 9034.923837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9034.945328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9039.539544] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9050.337808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9050.535847] Lustre: 297336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9050.603623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9050.607599] Lustre: Skipped 1 previous similar message [ 9050.683826] Lustre: lustre-MDT0001: new disk, initializing [ 9050.798433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9050.851115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9050.876225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9056.043579] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9065.525799] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9071.052398] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9078.352803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9078.689538] Lustre: lustre-OST2710: new disk, initializing [ 9078.695469] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9078.704870] Lustre: 299303:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9078.866412] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9085.451723] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9085.502838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9085.511870] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9085.636986] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9095.907463] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9103.474922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9103.640644] Lustre: lustre-OST03e8: new disk, initializing [ 9103.642908] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9103.647914] Lustre: 300368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9103.741257] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9110.235853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9110.260566] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9110.364607] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9111.382934] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9122.057765] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9131.311739] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9131.670972] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9137.143200] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9137.486659] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9142.389562] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9142.576763] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9147.367298] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9147.687511] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9153.507172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9153.534044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9153.547270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9156.576721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9156.579231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9156.599522] Lustre: Skipped 1 previous similar message [ 9156.607383] Lustre: Skipped 1 previous similar message [ 9158.032382] Lustre: server umount lustre-MDT0000 complete [ 9161.481496] LustreError: 297327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650583 with bad export cookie 2043105736201201605 [ 9161.485463] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9161.504572] LustreError: 297327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9161.705949] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9161.716965] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9161.722853] Lustre: Skipped 2 previous similar messages [ 9161.740378] Lustre: Skipped 3 previous similar messages [ 9162.125484] Lustre: server umount lustre-MDT0001 complete [ 9167.896799] Lustre: server umount lustre-OST2710 complete [ 9173.127861] Lustre: server umount lustre-OST03e8 complete [ 9192.264702] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 9202.972374] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9242.731557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9242.948580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9242.987828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9243.160975] Lustre: lustre-MDT0000: new disk, initializing [ 9243.327223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9243.348595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9248.175677] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9259.340949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.493398] Lustre: 305616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9259.555597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9259.559572] Lustre: Skipped 1 previous similar message [ 9259.627408] Lustre: lustre-MDT0001: new disk, initializing [ 9259.725027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9259.750616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9259.765585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9264.694257] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9273.408141] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9279.566714] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9286.044592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9286.258871] Lustre: lustre-OST0000: new disk, initializing [ 9286.261290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9286.265951] Lustre: 307582:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9286.345859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9292.188202] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9294.972055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9294.990559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9295.216954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9302.376466] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9308.114898] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9308.282594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9312.747193] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9312.987389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9315.815064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9315.820522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9315.826177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9315.832844] Lustre: Skipped 1 previous similar message [ 9315.839953] LustreError: Skipped 1 previous similar message [ 9320.934203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9320.951665] Lustre: Skipped 2 previous similar messages [ 9321.578942] Lustre: server umount lustre-OST0000 complete [ 9326.051214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9326.071582] Lustre: Skipped 1 previous similar message [ 9331.168843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9331.180385] Lustre: Skipped 3 previous similar messages [ 9340.383552] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9340.926571] Lustre: server umount lustre-MDT0000 complete [ 9341.413911] LustreError: 305621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9341.434953] LustreError: 305621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9345.384163] LustreError: 305609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650767 with bad export cookie 2043105736201203285 [ 9345.389157] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9345.393675] LustreError: 305609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9345.888746] Lustre: server umount lustre-MDT0001 complete [ 9355.790271] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:52:55 (1786650775) [ 9357.423691] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9359.166934] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:52:59 (1786650779) [ 9364.014855] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9393.614690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9393.976267] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9394.014494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9394.286657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9399.118209] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9409.205504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9409.272248] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9409.295751] Lustre: 311441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9415.321510] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9438.897063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9439.239971] Lustre: Denying initial registration attempt from nid 192.168.202.146@tcp, specified as failover [ 9439.249397] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9439.264664] LustreError: 312896:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9439.270844] LustreError: 312896:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9439.278218] LustreError: 312896:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9439.397646] Lustre: server umount lustre-OST0000 complete [ 9439.400070] LustreError: 312896:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9446.881632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9446.893290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9446.898568] Lustre: Skipped 1 previous similar message [ 9446.913560] Lustre: Skipped 3 previous similar messages [ 9452.807899] Lustre: server umount lustre-MDT0000 complete [ 9456.642737] LustreError: 311434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650879 with bad export cookie 2043105736201204167 [ 9456.654583] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9456.671272] LustreError: 311434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9456.920943] Lustre: server umount lustre-MDT0001 complete [ 9467.966978] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 9471.594887] Key type lgssc unregistered [ 9472.033578] LNet: 314022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9472.054111] LNetError: 314022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9472.081583] LNet: Removed LNI 192.168.202.146@tcp [ 9473.015318] Key type .llcrypt unregistered [ 9473.018309] Key type ._llcrypt unregistered [ 9484.396742] Key type ._llcrypt registered [ 9484.399591] Key type .llcrypt registered [ 9484.492081] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9485.608045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9485.677124] alg: No test for adler32 (adler32-zlib) [ 9486.963453] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 9487.241463] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 9489.015196] Key type lgssc registered [ 9490.465692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9546.611599] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9554.614426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9554.625584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9555.989738] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9556.026366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9556.230385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9560.764716] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9569.825438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9569.929551] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9569.999510] Lustre: 316788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9570.267687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9576.020924] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9584.821719] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9592.694729] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9612.593053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9613.113657] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9613.498202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9621.271624] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9629.893729] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9633.773591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9633.787512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9633.811803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9634.293487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9634.313475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9634.363220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9634.374786] Lustre: Skipped 1 previous similar message [ 9638.774804] Lustre: server umount lustre-OST0000 complete [ 9644.513077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9644.523725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9644.526432] Lustre: Skipped 1 previous similar message [ 9644.543798] Lustre: Skipped 1 previous similar message [ 9649.846637] Lustre: server umount lustre-MDT0000 complete [ 9654.024818] LustreError: 317538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786651076 with bad export cookie 15099124447330296848 [ 9654.037603] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9654.510691] Lustre: server umount lustre-MDT0001 complete [ 9669.556955] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 9673.394903] Key type lgssc unregistered [ 9673.811181] LNet: 320366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9673.824493] LNetError: 320366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9673.845752] LNet: Removed LNI 192.168.202.146@tcp [ 9675.060696] Key type .llcrypt unregistered [ 9675.064225] Key type ._llcrypt unregistered [ 9702.763965] Key type ._llcrypt registered [ 9702.768855] Key type .llcrypt registered [ 9702.891905] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9704.529582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9704.562215] alg: No test for adler32 (adler32-zlib) [ 9705.737634] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 9705.976462] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 9707.615183] Key type lgssc registered [ 9708.907768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9717.787799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9717.815989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9719.263923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9724.703169] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9729.509403] LustreError: 321780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9729.532598] LustreError: 321780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9734.634186] LustreError: 321781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9734.943573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9735.258521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9740.570825] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9748.039901] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9754.079805] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9763.591634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9763.964734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9771.760381] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9780.306883] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9790.978329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9790.986526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9791.007772] Lustre: Skipped 2 previous similar messages [ 9791.037239] Lustre: Skipped 2 previous similar messages [ 9792.361753] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 9794.726186] Lustre: server umount lustre-MDT0000 complete [ 9795.420110] LustreError: 321777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9798.833749] LustreError: 321763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786651221 with bad export cookie 1499125571178106010 [ 9798.861822] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9800.539790] LustreError: 321777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9800.571950] LustreError: 321777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9800.573946] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping) [ 9801.195145] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9803.233755] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9803.239657] Lustre: Skipped 1 previous similar message [ 9805.432898] Lustre: server umount lustre-MDT0001 complete [ 9811.907449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9822.949652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9823.657266] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x14cdf6c3a7c6aaa0 [ 9823.691099] Lustre: MGC192.168.202.146@tcp: Connection restored to 0@lo (at 0@lo) [ 9824.234400] LustreError: 323614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9824.256388] LustreError: 323614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9824.366232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9824.445225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9829.515782] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9832.545530] LustreError: 323615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9832.550207] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9832.570081] LustreError: 323615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9839.122833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9839.599954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9844.750642] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9844.804771] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9853.160043] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9855.543361] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9861.742755] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9863.568301] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9870.307411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9870.309589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9870.324532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9880.556729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9880.565108] Lustre: Skipped 4 previous similar messages [ 9884.127198] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9884.318803] Lustre: server umount lustre-OST0000 complete [ 9890.786268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9890.800281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9890.803356] Lustre: Skipped 2 previous similar messages [ 9894.295097] Lustre: server umount lustre-MDT0000 complete [ 9895.906572] LustreError: 325380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9895.926576] LustreError: 325380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9898.438778] LustreError: 321761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786651320 with bad export cookie 1499125571178113696 [ 9898.447056] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9898.451927] LustreError: 321761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9898.767628] Lustre: server umount lustre-MDT0001 complete [ 9909.248236] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 9912.977974] Key type lgssc unregistered [ 9913.268871] LNet: 328070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9913.274963] LNetError: 328070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9913.286981] LNet: Removed LNI 192.168.202.146@tcp [ 9914.203267] Key type .llcrypt unregistered [ 9914.206813] Key type ._llcrypt unregistered [ 9943.117534] Key type ._llcrypt registered [ 9943.120118] Key type .llcrypt registered [ 9943.229825] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 9944.851131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9944.937886] alg: No test for adler32 (adler32-zlib) [ 9945.866077] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 9946.006408] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 9947.663838] Key type lgssc registered [ 9949.154393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9957.263736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9957.294971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9958.901615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9964.478187] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9967.011626] LustreError: 329488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9967.053629] LustreError: 329488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9969.059415] LustreError: 329489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9974.243047] LustreError: 329488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9975.379181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9975.811862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9981.093558] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 9989.924069] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9996.755968] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10001.376975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10001.382222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10001.395982] Lustre: Skipped 1 previous similar message [10001.404755] Lustre: Skipped 1 previous similar message [10002.465378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10002.476319] Lustre: Skipped 1 previous similar message [10007.405583] Lustre: server umount lustre-MDT0000 complete [10007.535438] LustreError: 329489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10007.561143] LustreError: 329489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10011.991257] LustreError: 329470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786651434 with bad export cookie 4020864212940606675 [10012.006626] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10012.017129] LustreError: 329470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10012.470538] Lustre: server umount lustre-MDT0001 complete [10029.728156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10030.181918] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10030.236081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10030.522387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10036.302786] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10046.146212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10046.375125] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10046.415673] Lustre: 332078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10046.752983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10050.861547] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10061.724045] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10069.731885] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10081.321113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10081.463178] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10081.486079] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10081.493681] LustreError: 333964:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10081.500827] LustreError: 333964:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10081.506520] LustreError: 333964:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10081.563437] Lustre: server umount lustre-OST0000 complete [10081.566801] LustreError: 333964:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10092.417451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10092.677482] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10092.915557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10098.865412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10100.024890] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10108.987924] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10117.716356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10117.885884] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10118.011950] Lustre: lustre-OST0001: new disk, initializing [10118.020171] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10118.028667] Lustre: 335508:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10118.219971] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10124.523877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10124.537878] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10124.731295] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10127.000053] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10136.767680] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10145.255509] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10145.275234] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10145.286460] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10145.287727] Lustre: Skipped 1 previous similar message [10145.298870] Lustre: Skipped 1 previous similar message [10147.048619] Lustre: server umount lustre-OST0001 complete [10150.389568] LustreError: 334431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10150.405247] LustreError: 334431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10154.976284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10154.982317] LustreError: Skipped 1 previous similar message [10154.987116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10155.002289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10160.609200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10160.630081] Lustre: Skipped 3 previous similar messages [10160.636561] LustreError: 334429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10160.680043] LustreError: 334429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10165.215183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10165.439279] Lustre: server umount lustre-OST0000 complete [10169.824885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10169.834487] LustreError: Skipped 1 previous similar message [10169.839863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10169.849342] Lustre: Skipped 1 previous similar message [10169.854784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10175.791348] Lustre: server umount lustre-MDT0000 complete [10179.851688] LustreError: 332070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786651602 with bad export cookie 4020864212940607228 [10179.853182] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10179.866842] LustreError: 332070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10180.353609] Lustre: server umount lustre-MDT0001 complete [10192.296211] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [10195.863333] Key type lgssc unregistered [10196.211386] LNet: 337472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10196.223539] LNetError: 337472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10196.243621] LNet: Removed LNI 192.168.202.146@tcp [10197.178368] Key type .llcrypt unregistered [10197.184752] Key type ._llcrypt unregistered [10276.530825] Key type ._llcrypt registered [10276.534838] Key type .llcrypt registered [10276.682047] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [10293.422471] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [10294.926415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10294.972701] alg: No test for adler32 (adler32-zlib) [10296.051709] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [10296.301442] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [10298.001202] Key type lgssc registered [10299.146333] Lustre: Echo OBD driver; http://www.lustre.org/ [10335.142440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10335.178692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10336.495587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10336.554467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10336.709912] Lustre: lustre-MDT0000: new disk, initializing [10336.873172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10336.902960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10342.478266] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10353.620180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10353.790307] Lustre: 342410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10353.851793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10353.856924] Lustre: Skipped 1 previous similar message [10353.961652] Lustre: lustre-MDT0001: new disk, initializing [10354.126714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10354.195526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10354.218665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10360.502822] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10373.393359] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10379.490354] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10386.088627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10386.326611] Lustre: lustre-OST0000: new disk, initializing [10386.330522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10386.336271] Lustre: 344380:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10386.415422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10393.269029] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10394.189388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10394.200304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10394.300624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10403.280366] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10409.271529] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10409.534192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10415.263541] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10415.539440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10420.206884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10420.211446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10420.241079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10420.244519] Lustre: Skipped 1 previous similar message [10423.833746] Lustre: server umount lustre-OST0000 complete [10430.440764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10430.462888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10430.472144] Lustre: Skipped 1 previous similar message [10430.485817] Lustre: Skipped 2 previous similar messages [10435.553474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10435.561925] Lustre: Skipped 1 previous similar message [10439.663514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10439.668691] Lustre: Skipped 1 previous similar message [10442.211553] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10442.511627] Lustre: server umount lustre-MDT0000 complete [10446.847974] LustreError: 344379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786651869 with bad export cookie 117318538144447040 [10446.851220] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10446.856171] LustreError: 344379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10447.227866] Lustre: server umount lustre-MDT0001 complete [10456.938360] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 16:11:16 (1786651876) [10465.350210] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 16:11:25 (1786651885) [10474.416954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10474.882604] LustreError: 347139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10474.899474] LustreError: 347139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10474.985123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10478.767488] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10480.097909] LustreError: 347140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10485.227403] LustreError: 347139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10487.429720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10487.909646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10493.223869] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10501.039812] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10507.594299] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10517.999883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10518.706313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10526.949573] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10535.304298] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10543.019896] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10545.124319] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10546.903330] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10548.614729] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10551.040426] Lustre: Failing over lustre-MDT0000 [10551.332318] Lustre: server umount lustre-MDT0000 complete [10552.692208] LustreError: 347135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10553.825076] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.838943] Lustre: Skipped 2 previous similar messages [10557.779314] LustreError: 347135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10557.794069] LustreError: 347135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10568.017532] LustreError: 349456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10568.035456] LustreError: 349456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10569.918933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10570.079465] Lustre: 340635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786651976/real 1786651976] req@00000000c8c4024c x1873440111929344/t0(0) o400->MGC192.168.202.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786651992 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10570.085615] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10570.374521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10570.433879] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10574.167416] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10574.860038] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10575.855694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10575.894627] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10575.960991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10584.406331] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10585.939683] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10588.805525] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10590.738417] Lustre: Failing over lustre-MDT0000 [10590.934281] Lustre: server umount lustre-MDT0000 complete [10591.202317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10591.213540] LustreError: Skipped 1 previous similar message [10591.219190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10591.232259] Lustre: Skipped 2 previous similar messages [10591.256619] LustreError: 347134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10591.272677] LustreError: 347134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10607.058648] Lustre: 340635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786652013/real 1786652013] req@000000004492c289 x1873440111947392/t0(0) o400->MGC192.168.202.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786652029 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10607.108604] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10613.458329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10617.314690] LustreError: 340633:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006c5dc522 x1873440111953408/t0(0) o250->MGC192.168.202.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10617.703482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10617.788994] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10621.275922] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10622.201233] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10622.969335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10622.984494] Lustre: Skipped 2 previous similar messages [10623.018960] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10623.109365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10632.932638] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10634.544743] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10643.427525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10643.428388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10643.464149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10647.504604] Lustre: server umount lustre-MDT0000 complete [10648.546146] LustreError: 347867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10648.569833] LustreError: 347867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [10651.905380] LustreError: 347121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786652074 with bad export cookie 117318538144449749 [10651.927820] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10651.931347] LustreError: 347121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10653.060957] Lustre: server umount lustre-MDT0001 complete [10666.801869] Lustre: server umount lustre-OST0000 complete [10684.488787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10684.875582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10690.103918] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10699.378959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10704.927628] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10712.824974] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10719.498354] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10726.965676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10730.378511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10734.043106] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10742.856813] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10755.039969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10755.055202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10755.062755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10755.062764] Lustre: Skipped 2 previous similar messages [10755.107172] Lustre: Skipped 3 previous similar messages [10759.626140] Lustre: server umount lustre-OST0000 complete [10770.406656] LustreError: 353481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10770.434485] LustreError: 353481:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10770.572506] Lustre: server umount lustre-MDT0000 complete [10774.291080] LustreError: 353462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786652196 with bad export cookie 117318538144450498 [10774.300975] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10774.304289] LustreError: 353462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10774.785136] Lustre: server umount lustre-MDT0001 complete [10791.575608] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [10795.198215] Key type lgssc unregistered [10795.601752] LNet: 357016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10795.609032] LNetError: 357016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10795.666256] LNet: Removed LNI 192.168.202.146@tcp [10796.958377] Key type .llcrypt unregistered [10796.960344] Key type ._llcrypt unregistered [10828.643459] Key type ._llcrypt registered [10828.646401] Key type .llcrypt registered [10828.761520] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [10830.176171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10830.536860] alg: No test for adler32 (adler32-zlib) [10831.755808] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [10831.946540] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [10833.639161] Key type lgssc registered [10835.131527] Lustre: Echo OBD driver; http://www.lustre.org/ [10843.757876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10843.780143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10845.463546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10849.127997] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10854.882840] LustreError: 358535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10854.923469] LustreError: 358535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10860.012604] LustreError: 358536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10860.880464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.516325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10869.529753] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10878.845226] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10886.539445] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10893.682232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10894.036209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10901.314941] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10910.365686] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10914.634704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10930.152142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10930.153504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10930.158043] Lustre: Skipped 2 previous similar messages [10930.162501] Lustre: Skipped 1 previous similar message [10935.148168] Lustre: server umount lustre-MDT0000 complete [10935.276058] LustreError: 359265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.286464] LustreError: 359265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10938.996315] LustreError: 358515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786652361 with bad export cookie 5011625566608216811 [10939.007477] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10939.011230] LustreError: 358515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10939.406654] Lustre: server umount lustre-MDT0001 complete [10953.722813] Lustre: server umount lustre-OST0000 complete [10968.051617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10968.653246] LustreError: 362279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10968.732665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10974.115257] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [10974.184709] LustreError: 362280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10984.419786] LustreError: 362279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10984.433298] LustreError: 362279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10984.575903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10984.957046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10990.225227] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11000.163866] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11008.828566] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11016.545267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11016.910563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11024.146733] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11032.718681] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11039.414456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11060.207027] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11060.210445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11060.226423] Lustre: Skipped 1 previous similar message [11063.235042] Lustre: server umount lustre-MDT0000 complete [11065.313054] LustreError: 362280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11065.335882] LustreError: 362280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11067.232749] LustreError: 362259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786652489 with bad export cookie 5011625566608217924 [11067.250790] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11067.252171] LustreError: 362259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11067.722843] Lustre: server umount lustre-MDT0001 complete [11082.389936] Lustre: server umount lustre-OST0000 complete [11105.604317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11106.540339] LustreError: 366212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11106.678719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11111.665631] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11121.147394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11126.867826] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11135.893921] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11142.287759] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11150.482947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11150.732419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11150.739206] Lustre: Skipped 1 previous similar message [11151.804151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11158.157746] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11168.174604] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11177.444586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11177.456050] Lustre: Skipped 1 previous similar message [11177.458636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11177.460576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11177.460583] Lustre: Skipped 2 previous similar messages [11180.594917] Lustre: server umount lustre-OST0000 complete [11187.681750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11187.683727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11187.706067] Lustre: Skipped 1 previous similar message [11187.727292] Lustre: Skipped 2 previous similar messages [11191.249099] Lustre: server umount lustre-MDT0000 complete [11192.803633] LustreError: 366213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11192.825176] LustreError: 366213:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11196.196886] LustreError: 366194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786652618 with bad export cookie 5011625566608218946 [11196.211668] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11196.214674] LustreError: 366194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11196.633484] Lustre: server umount lustre-MDT0001 complete [11209.658308] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [11213.256832] Key type lgssc unregistered [11213.668711] LNet: 369750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11213.672851] LNetError: 369750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11213.706742] LNet: Removed LNI 192.168.202.146@tcp [11215.023994] Key type .llcrypt unregistered [11215.027329] Key type ._llcrypt unregistered [11247.332566] Key type ._llcrypt registered [11247.334026] Key type .llcrypt registered [11247.447873] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [11248.776930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11249.023305] alg: No test for adler32 (adler32-zlib) [11250.327476] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [11250.630879] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [11252.359834] Key type lgssc registered [11253.952910] Lustre: Echo OBD driver; http://www.lustre.org/ [11263.679728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11263.692150] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11263.705445] LustreError: 371191:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11263.729323] LustreError: 371191:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11263.738135] LustreError: 371191:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11263.753843] LustreError: 371191:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11263.772648] LustreError: 371191:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11273.658497] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11273.675688] LustreError: 371588:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11273.697187] LustreError: 371588:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11273.704162] LustreError: 371588:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11273.710827] LustreError: 371588:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11273.717929] LustreError: 371588:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11301.162773] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [11305.195284] Key type lgssc unregistered [11305.631746] LNet: 372597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11305.636748] LNetError: 372597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11305.659391] LNet: Removed LNI 192.168.202.146@tcp [11306.765714] Key type .llcrypt unregistered [11306.772164] Key type ._llcrypt unregistered [11330.923938] Key type ._llcrypt registered [11330.930971] Key type .llcrypt registered [11331.085844] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [11349.214892] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [11350.231357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11350.270157] alg: No test for adler32 (adler32-zlib) [11351.618233] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [11351.917612] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [11353.631212] Key type lgssc registered [11354.959742] Lustre: Echo OBD driver; http://www.lustre.org/ [11396.953686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11397.003040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11398.461792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11398.525712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11398.659959] Lustre: lustre-MDT0000: new disk, initializing [11398.771873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11398.797281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11402.890476] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11415.639312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11415.904444] Lustre: 375755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11415.980544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11415.988653] Lustre: Skipped 1 previous similar message [11416.152457] Lustre: lustre-MDT0001: new disk, initializing [11416.405456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11416.480212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11416.500919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11423.961856] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11435.703420] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11441.906724] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11450.529493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11450.959297] Lustre: lustre-OST0000: new disk, initializing [11450.967427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11450.978514] Lustre: 377723:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11451.154715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11457.661196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11457.674655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11457.796852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11458.760222] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11469.725117] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11475.591272] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11475.926316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11481.446989] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11481.761505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11488.746956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11488.771105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11488.772970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11488.787569] Lustre: Skipped 1 previous similar message [11488.799926] LustreError: Skipped 1 previous similar message [11490.376307] Lustre: server umount lustre-OST0000 complete [11498.465140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11498.475215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11498.486992] Lustre: Skipped 1 previous similar message [11498.503244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11501.884640] Lustre: server umount lustre-MDT0000 complete [11505.858645] LustreError: 375746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786652928 with bad export cookie 9800969013204102901 [11505.865977] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11505.877602] LustreError: 375746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11506.262893] Lustre: server umount lustre-MDT0001 complete [11515.607255] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11517.381626] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 16:28:57 (1786652937) [11525.071305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11525.616278] LustreError: 380185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11525.633801] LustreError: 380185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11525.686084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11530.734263] LustreError: 380186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11531.114188] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11535.841579] LustreError: 380185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11540.234525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11540.595254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11546.309789] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11554.627986] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11561.499986] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11569.601363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11569.968895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11576.805362] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11585.947377] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11593.364890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11593.677970] Lustre: lustre-OST0001: new disk, initializing [11593.681661] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11593.693521] Lustre: 383024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11593.912328] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11600.539159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11600.555204] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11600.689605] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11603.321716] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11616.287581] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11621.357608] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11621.369192] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11621.378069] Lustre: Skipped 1 previous similar message [11621.395050] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11621.405287] Lustre: Skipped 2 previous similar messages [11626.491771] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11626.496915] Lustre: Skipped 2 previous similar messages [11626.659662] Lustre: server umount lustre-OST0001 complete [11628.383061] LustreError: 382010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11636.709293] LustreError: 382009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11636.730323] LustreError: 382009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11646.949840] LustreError: 383285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11646.950842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11646.968272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11647.001604] LustreError: 383285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11647.070741] Lustre: Skipped 2 previous similar messages [11657.189341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11657.200150] Lustre: Skipped 4 previous similar messages [11659.231833] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11659.517189] Lustre: server umount lustre-OST0000 complete [11667.425556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11671.665599] Lustre: server umount lustre-MDT0000 complete [11672.544704] LustreError: 380186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11672.578164] LustreError: 380186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11676.270743] LustreError: 380165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786653098 with bad export cookie 9800969013204103783 [11676.282753] LustreError: 380165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11676.282854] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11682.711800] Lustre: server umount lustre-MDT0001 complete [11696.158853] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [11700.415361] Key type lgssc unregistered [11700.837234] LNet: 385017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11700.848550] LNetError: 385017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11700.877139] LNet: Removed LNI 192.168.202.146@tcp [11702.211829] Key type .llcrypt unregistered [11702.218854] Key type ._llcrypt unregistered [11761.381729] Key type ._llcrypt registered [11761.384569] Key type .llcrypt registered [11761.487506] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [11763.657868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11763.706974] alg: No test for adler32 (adler32-zlib) [11764.940970] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [11765.331656] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [11767.087241] Key type lgssc registered [11768.863296] Lustre: Echo OBD driver; http://www.lustre.org/ [11779.517166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11779.565184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11781.101138] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11781.138763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11781.581322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11788.577126] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11798.292377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11798.483753] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11798.563516] Lustre: 387821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11798.861517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11803.831192] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11812.181660] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11818.098640] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11822.049124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11822.071897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11822.077368] Lustre: Skipped 1 previous similar message [11826.930591] Lustre: server umount lustre-MDT0000 complete [11830.681864] LustreError: 387816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786653253 with bad export cookie 2343061319637347225 [11830.690525] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11830.695986] LustreError: 387816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11831.199392] Lustre: server umount lustre-MDT0001 complete [11842.908219] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 16:34:23 (1786653263) [11849.936727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11850.559563] LustreError: 390333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11850.595139] LustreError: 390333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11850.648418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11855.854154] LustreError: 390334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11856.339743] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11860.963720] LustreError: 390333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11865.896508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11866.083329] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11866.229733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11871.699572] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11880.598902] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11887.729618] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11895.180363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11895.394273] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11895.659512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11902.682136] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11910.551449] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11918.611206] Lustre: Permanently deactivating lustre-OST0000 [11918.631426] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11928.640741] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11930.279212] LustreError: 393366:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11930.295571] LustreError: 393366:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11930.304300] LustreError: 393366:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11933.338558] LustreError: 393414:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11933.350991] LustreError: 393414:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11941.818825] Lustre: server umount lustre-OST0000 complete [11944.924234] LustreError: 393665:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11944.935779] LustreError: 393665:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11948.006197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11948.018133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11948.052273] Lustre: Skipped 1 previous similar message [11948.082544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11951.074359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11951.075451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11956.194206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11956.221871] Lustre: Skipped 2 previous similar messages [11958.319568] Lustre: server umount lustre-MDT0000 complete [11961.312620] LustreError: 390334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11961.337693] LustreError: 390334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11962.267276] LustreError: 390315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786653384 with bad export cookie 2343061319637347806 [11962.280579] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11962.290589] LustreError: 390315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11971.941455] Lustre: server umount lustre-MDT0001 complete [11981.765634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11986.407910] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [11989.382869] LustreError: 394825:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11997.079957] Lustre: 395164:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.146@tcp [11999.516334] Lustre: server umount MGS complete [12011.084045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12011.707375] LustreError: 395888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12011.725554] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12011.727570] Lustre: Skipped 1 previous similar message [12011.776149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12016.863434] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12022.240713] LustreError: 395889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12022.255401] LustreError: 395889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12026.944237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12027.141277] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12032.068917] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12039.853492] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12046.157562] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12048.062172] Lustre: Permanently reactivating lustre-OST0000 [12048.078820] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12048.087523] Lustre: Skipped 2 previous similar messages [12076.000530] LustreError: 397667:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12082.927224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12083.522785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12083.525829] Lustre: Skipped 1 previous similar message [12092.555796] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12101.555459] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12109.529727] Lustre: server umount lustre-OST0000 complete [12114.405500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12114.414061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12114.430650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12127.711174] Lustre: 387143:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786653533/real 1786653533] req@00000000ca029a9f x1873441652311552/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786653549 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12127.789885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12127.831235] Lustre: Skipped 1 previous similar message [12128.225391] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12128.515663] Lustre: server umount lustre-MDT0000 complete [12131.811041] Lustre: 387146:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786653538/real 1786653538] req@00000000a11a6beb x1873441652314112/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786653554 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12132.505639] LustreError: 395870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786653555 with bad export cookie 2343061319637348856 [12132.551318] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12133.014670] Lustre: server umount lustre-MDT0001 complete [12146.416263] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [12149.339461] Key type lgssc unregistered [12149.624818] LNet: 400255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12149.634504] LNetError: 400255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12150.699819] LNet: Removed LNI 192.168.202.146@tcp [12151.932698] Key type .llcrypt unregistered [12151.944881] Key type ._llcrypt unregistered [12172.010644] Key type ._llcrypt registered [12172.012547] Key type .llcrypt registered [12172.087885] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [12187.952527] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [12189.052342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12189.248962] alg: No test for adler32 (adler32-zlib) [12190.388876] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [12190.712743] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [12192.727337] Key type lgssc registered [12194.795104] Lustre: Echo OBD driver; http://www.lustre.org/ [12244.937676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12244.994251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12246.585355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12246.706365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12246.998827] Lustre: lustre-MDT0000: new disk, initializing [12247.238699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12247.296313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12254.979211] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12268.259299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12268.539057] Lustre: 403414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12268.646879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12268.662358] Lustre: Skipped 1 previous similar message [12268.786472] Lustre: lustre-MDT0001: new disk, initializing [12268.959832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12269.033467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12269.043105] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12274.186719] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12284.980700] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12291.669342] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12299.545196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12299.784351] Lustre: lustre-OST0000: new disk, initializing [12299.787373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12299.792838] Lustre: 405386:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12299.889667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12307.427866] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12310.123546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12310.130780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12310.286831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12318.356464] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12323.849898] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12324.125962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12329.020616] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12329.223932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12336.108108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12336.114068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12336.126126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12337.753917] Lustre: server umount lustre-OST0000 complete [12345.835048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12345.842540] LustreError: Skipped 1 previous similar message [12345.849103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12345.859508] Lustre: Skipped 1 previous similar message [12345.870313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12345.876569] Lustre: Skipped 1 previous similar message [12348.971689] Lustre: server umount lustre-MDT0000 complete [12353.263835] LustreError: 403408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786653775 with bad export cookie 14030130550150048910 [12353.285978] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12354.058630] Lustre: server umount lustre-MDT0001 complete [12363.879851] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12365.516784] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 16:43:05 (1786653785) [12373.170743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12373.678545] LustreError: 407846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12373.709767] LustreError: 407846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12373.787951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12379.114936] LustreError: 407847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12379.647331] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12384.232251] LustreError: 407846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12389.354926] LustreError: 407847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12389.567620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12390.025122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12395.952968] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12405.092450] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12411.367895] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12419.573098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12420.029068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12426.782852] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12435.949114] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12438.477332] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12446.338851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12453.859424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12453.877211] Lustre: Skipped 1 previous similar message [12453.899959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12453.902375] Lustre: Skipped 2 previous similar messages [12458.469680] Lustre: server umount lustre-OST0000 complete [12464.097406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12464.111318] Lustre: Skipped 1 previous similar message [12464.122859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12464.132115] Lustre: Skipped 1 previous similar message [12469.216283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12469.838625] Lustre: server umount lustre-MDT0000 complete [12473.890984] LustreError: 407827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786653896 with bad export cookie 14030130550150049785 [12473.898735] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12473.906638] LustreError: 407827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12474.205800] Lustre: server umount lustre-MDT0001 complete [12487.282799] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [12490.390806] Key type lgssc unregistered [12490.692653] LNet: 411578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12490.699390] LNetError: 411578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12491.749353] LNet: Removed LNI 192.168.202.146@tcp [12492.666374] Key type .llcrypt unregistered [12492.668253] Key type ._llcrypt unregistered [12521.302575] Key type ._llcrypt registered [12521.304843] Key type .llcrypt registered [12521.435229] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [12521.779329] Key type .llcrypt unregistered [12521.781361] Key type ._llcrypt unregistered [12542.087177] Key type ._llcrypt registered [12542.090564] Key type .llcrypt registered [12542.280756] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [12543.842785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12543.868740] alg: No test for adler32 (adler32-zlib) [12545.081897] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [12545.380268] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [12547.087280] Key type lgssc registered [12548.729124] Lustre: Echo OBD driver; http://www.lustre.org/ [12558.452145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12558.512862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12560.274325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12565.949297] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12569.574051] LustreError: 413795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12569.616019] LustreError: 413795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12574.698707] LustreError: 413796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12576.372147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12576.760277] LustreError: 413795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12576.807901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12581.859221] LustreError: 413796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12581.891970] LustreError: 413796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12584.970587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12585.512604] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12590.055311] LustreError: 413795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12590.099550] LustreError: 413795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12594.807789] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12600.300645] LustreError: 413796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12600.323145] LustreError: 413796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12604.796112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12605.537471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12613.772361] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12625.600660] Lustre: server umount lustre-OST0000 complete [12631.010915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12631.022546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12631.045751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12631.068966] Lustre: Skipped 1 previous similar message [12636.134296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12636.150250] Lustre: Skipped 2 previous similar messages [12640.353580] Lustre: server umount lustre-MDT0000 complete [12641.253958] LustreError: 413796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12641.288469] LustreError: 413796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12642.271120] Lustre: 413106:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786654048/real 1786654048] req@000000005ed9aa6c x1873442470128768/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786654064 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12642.309543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12644.625990] LustreError: 413775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786654067 with bad export cookie 10484869303945779436 [12644.638812] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12644.646281] LustreError: 413775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12645.249821] Lustre: server umount lustre-MDT0001 complete [12658.027709] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [12661.881707] Key type lgssc unregistered [12662.294289] LNet: 416949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12662.319612] LNetError: 416949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12662.336303] LNet: Removed LNI 192.168.202.146@tcp [12663.655260] Key type .llcrypt unregistered [12663.656979] Key type ._llcrypt unregistered [12692.962829] Key type ._llcrypt registered [12692.968436] Key type .llcrypt registered [12693.106653] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [12694.605643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12694.825526] alg: No test for adler32 (adler32-zlib) [12695.905783] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [12696.134913] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [12697.807190] Key type lgssc registered [12698.673674] Lustre: Echo OBD driver; http://www.lustre.org/ [12708.491946] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12708.526829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12724.127620] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12729.311641] LustreError: 418342:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.146@tcp: failed processing log, type 4: rc = -110 [12756.063653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12763.568837] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12772.774609] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12779.786452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12780.297758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12780.347220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12784.988619] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12785.638373] LustreError: 418367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12785.648528] LustreError: 418367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12790.764205] LustreError: 418368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12790.793477] LustreError: 418368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12795.880592] LustreError: 419380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12795.917440] LustreError: 419380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12796.107111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12796.763491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12802.177934] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12811.853914] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12818.766888] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12831.714099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12831.730426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12831.758215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12832.741242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12832.753279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12832.786095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12834.816682] Lustre: server umount lustre-OST0000 complete [12842.978682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12842.986211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12842.991206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12843.012137] Lustre: Skipped 1 previous similar message [12843.028884] Lustre: Skipped 2 previous similar messages [12848.098185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12848.113255] Lustre: Skipped 1 previous similar message [12853.220433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12853.240763] Lustre: Skipped 1 previous similar message [12854.242652] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12856.474583] Lustre: server umount lustre-MDT0000 complete [12858.346942] LustreError: 419379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12858.364278] LustreError: 419379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12861.163425] LustreError: 418348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786654283 with bad export cookie 14492758956156519790 [12861.183914] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12861.190854] LustreError: 418348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12862.044741] Lustre: server umount lustre-MDT0001 complete [12876.329537] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [12879.751495] Key type lgssc unregistered [12880.112812] LNet: 421920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12880.151816] LNetError: 421920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12880.177821] LNet: Removed LNI 192.168.202.146@tcp [12881.524715] Key type .llcrypt unregistered [12881.527483] Key type ._llcrypt unregistered [12910.082604] Key type ._llcrypt registered [12910.084283] Key type .llcrypt registered [12910.209626] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [12911.892272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12911.962106] alg: No test for adler32 (adler32-zlib) [12913.242131] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [12913.646045] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [12915.443960] Key type lgssc registered [12916.855835] Lustre: Echo OBD driver; http://www.lustre.org/ [12926.425211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12926.445858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12928.146380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12933.212881] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12937.699214] LustreError: 423339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12937.728675] LustreError: 423339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12942.587193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12942.832242] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12943.096548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12948.913444] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12956.456791] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12963.676922] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12970.549532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12970.885288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12973.005604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12978.960360] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [12987.982903] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12993.515703] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12993.520974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12993.531414] Lustre: Skipped 1 previous similar message [12995.567232] Lustre: server umount lustre-MDT0000 complete [12995.953939] LustreError: 423336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12998.626412] LustreError: 423334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12998.660562] LustreError: 423334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12998.755569] LustreError: 425183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.46@tcp arrived at 1786654421 with bad export cookie 9801887907290810414 [12998.771405] LustreError: 425183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13000.981548] LustreError: 423319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786654423 with bad export cookie 9801887907290809511 [13001.006941] LustreError: 423319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13001.007172] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13003.745192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13003.746841] LustreError: 423336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13003.752313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13003.752322] Lustre: Skipped 1 previous similar message [13003.753701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13003.753708] Lustre: Skipped 2 previous similar messages [13003.798216] LustreError: 423336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13006.758206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13007.451536] Lustre: server umount lustre-OST0000 complete [13010.912945] LustreError: 423340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13010.958281] LustreError: 423340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13016.554570] Lustre: server umount lustre-MDT0001 complete [13034.872584] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13038.423572] Key type lgssc unregistered [13039.048157] LNet: 427046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13039.055806] LNetError: 427046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13039.096092] LNet: Removed LNI 192.168.202.146@tcp [13039.905246] Key type .llcrypt unregistered [13039.907386] Key type ._llcrypt unregistered [13071.303194] Key type ._llcrypt registered [13071.308675] Key type .llcrypt registered [13071.459842] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [13073.409920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13073.478519] alg: No test for adler32 (adler32-zlib) [13074.596497] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [13074.994829] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [13076.743830] Key type lgssc registered [13078.197591] Lustre: Echo OBD driver; http://www.lustre.org/ [13087.277610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13087.309276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13088.710027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13094.084586] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13098.979229] LustreError: 428466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13099.008515] LustreError: 428466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13103.718655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13104.083186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13109.867414] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13118.336906] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13125.358634] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13133.223944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13133.721888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13134.810219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13140.148556] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13148.212848] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13154.277453] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13154.294875] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13154.299704] Lustre: Skipped 1 previous similar message [13154.314066] Lustre: Skipped 1 previous similar message [13156.247288] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping) [13159.395330] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13159.749571] Lustre: server umount lustre-MDT0001 complete [13161.297135] LustreError: 428463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13164.513814] LustreError: 429406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13164.534047] LustreError: 429406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13169.634745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13169.639183] LustreError: 428466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13169.650095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13169.683141] Lustre: Skipped 1 previous similar message [13173.263375] Lustre: server umount lustre-OST0000 complete [13174.754937] LustreError: 429406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13177.970962] Lustre: server umount lustre-MDT0000 complete [13194.370098] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13197.708672] Key type lgssc unregistered [13198.031732] LNet: 432155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13198.043139] LNetError: 432155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13198.062216] LNet: Removed LNI 192.168.202.146@tcp [13198.855443] Key type .llcrypt unregistered [13198.858167] Key type ._llcrypt unregistered [13226.452781] Key type ._llcrypt registered [13226.459238] Key type .llcrypt registered [13226.638561] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [13242.542448] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [13243.671667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13243.774843] alg: No test for adler32 (adler32-zlib) [13244.793458] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [13244.977201] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [13246.656174] Key type lgssc registered [13247.790778] Lustre: Echo OBD driver; http://www.lustre.org/ [13294.166569] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [13310.778846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13310.804711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13312.064351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13312.091465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13312.175846] Lustre: lustre-MDT0000: new disk, initializing [13312.252553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13312.271204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13317.387180] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13335.490745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13335.852295] Lustre: lustre-OST0000: new disk, initializing [13335.866070] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13335.872596] Lustre: Skipped 1 previous similar message [13335.888327] Lustre: 437189:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13336.030200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13341.838623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13341.860067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13341.912211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13343.971942] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13354.250565] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13372.238669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13372.398116] Lustre: 436173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13372.475891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13372.600781] Lustre: lustre-MDT0001: new disk, initializing [13372.786057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13372.843327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13372.857512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13378.494295] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13390.227994] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13390.469574] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13395.966449] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13396.191279] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13402.080482] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13402.089190] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13402.102153] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13402.106066] Lustre: Skipped 1 previous similar message [13403.028563] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping) [13407.204749] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13407.208493] Lustre: Skipped 1 previous similar message [13407.714281] Lustre: server umount lustre-MDT0001 complete [13417.443021] LustreError: 436178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13417.450651] LustreError: 436178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13422.565702] LustreError: 436180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13422.601389] LustreError: 436180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13427.689143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13427.691082] LustreError: 436184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13427.711109] Lustre: Skipped 1 previous similar message [13427.741202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13431.710524] Lustre: server umount lustre-OST0000 complete [13432.800306] LustreError: 436183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13442.463365] Lustre: server umount lustre-MDT0000 complete [13457.052217] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13460.382264] Key type lgssc unregistered [13460.686464] LNet: 440954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13460.695555] LNetError: 440954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13461.733194] LNet: Removed LNI 192.168.202.146@tcp [13462.706475] Key type .llcrypt unregistered [13462.708230] Key type ._llcrypt unregistered [13503.472543] Key type ._llcrypt registered [13503.475201] Key type .llcrypt registered [13503.638494] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [13503.988506] Key type .llcrypt unregistered [13503.997994] Key type ._llcrypt unregistered [13521.892486] Key type ._llcrypt registered [13521.896367] Key type .llcrypt registered [13522.065430] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [13523.094231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13523.126940] alg: No test for adler32 (adler32-zlib) [13524.551692] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [13524.947251] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [13526.631205] Key type lgssc registered [13527.962961] Lustre: Echo OBD driver; http://www.lustre.org/ [13554.643893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13554.696296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13555.935884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13555.957587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13556.035582] Lustre: lustre-MDT0000: new disk, initializing [13556.132342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13556.147829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13560.561873] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13570.767898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13570.907287] Lustre: 444337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13570.984559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13570.988614] Lustre: Skipped 1 previous similar message [13571.076452] Lustre: lustre-MDT0001: new disk, initializing [13571.189757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13571.232457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13571.240465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13575.429806] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13583.941979] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13588.964516] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13595.551164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13595.828674] Lustre: lustre-OST0000: new disk, initializing [13595.832275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13595.841866] Lustre: 446304:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13595.949489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13601.991427] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13603.417073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13603.425594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13603.555181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13612.099896] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13619.178584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13619.181562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13619.195595] Lustre: Skipped 2 previous similar messages [13619.222423] Lustre: Skipped 2 previous similar messages [13624.290155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13624.294627] Lustre: Skipped 2 previous similar messages [13624.815282] Lustre: server umount lustre-MDT0000 complete [13629.392517] LustreError: 444331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786655051 with bad export cookie 1545840040568878985 [13629.401259] LustreError: 444331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13629.402417] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13629.419127] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13629.426429] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13629.681252] Lustre: server umount lustre-MDT0001 complete [13636.578061] Lustre: server umount lustre-OST0000 complete [13649.080268] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:04:29 (1786655069) [13656.813357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13657.245871] LustreError: 448332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13657.275821] LustreError: 448332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13657.357571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13661.160164] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13662.693586] LustreError: 448332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13666.786404] LustreError: 448333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13670.263088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13670.697741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13676.518637] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13685.073570] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13691.682696] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13711.993808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13712.310835] Lustre: Found index 0 for lustre-OST0000, updating log [13712.481131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13718.866971] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [13726.851362] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13727.858140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13738.467929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13738.478134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13738.484324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13738.521741] Lustre: Skipped 1 previous similar message [13744.522942] Lustre: server umount lustre-OST0000 complete [13760.992239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13761.003274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13761.013129] Lustre: Skipped 4 previous similar messages [13767.142534] LustreError: 448332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13767.447943] Lustre: server umount lustre-MDT0000 complete [13772.256569] LustreError: 448327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13772.280931] LustreError: 448327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13772.497725] LustreError: 448314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786655195 with bad export cookie 1545840040568880378 [13772.500461] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13772.505996] LustreError: 448314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13772.987866] Lustre: server umount lustre-MDT0001 complete [13775.848790] LNet: 1 local NIs in recovery (showing 1): 192.168.202.146@tcp [13783.841777] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:06:43 (1786655203) [14410.961542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14411.480847] LustreError: 454749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14411.567583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14416.864605] LustreError: 454748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14417.247049] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [14421.990572] LustreError: 454749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14426.322596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14426.676396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14431.989306] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [14440.697501] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14446.726383] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14466.079630] LNet: 1 local NIs in recovery (showing 1): 192.168.202.146@tcp [14466.236248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14466.533706] Lustre: Found index 0 for lustre-OST0000, updating log [14466.786694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14474.330254] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [14483.143583] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14485.591735] Lustre: server umount lustre-OST0000 complete [14490.984046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14490.997474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14491.000763] Lustre: Skipped 2 previous similar messages [14491.016901] Lustre: Skipped 5 previous similar messages [14492.069870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14492.075503] Lustre: Skipped 1 previous similar message [14494.113971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14494.125540] Lustre: Skipped 3 previous similar messages [14496.175000] LustreError: 454744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14496.210321] LustreError: 454744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14496.592282] Lustre: server umount lustre-MDT0000 complete [14500.862448] LustreError: 454730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786655923 with bad export cookie 1545840040568881470 [14500.864887] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14500.868978] LustreError: 454730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14501.408890] Lustre: server umount lustre-MDT0001 complete [14526.501305] LNet: 1 local NIs in recovery (showing 1): 192.168.202.146@tcp [14533.137702] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [14544.651374] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [14588.218939] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 17:20:08 (1786656008) [14609.818911] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14613.480329] Key type lgssc unregistered [14613.915834] LNet: 461366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14613.924257] LNetError: 461366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14726.566391] LNet: Removed LNI 192.168.202.146@tcp [14727.643215] Key type .llcrypt unregistered [14727.647384] Key type ._llcrypt unregistered [14747.743702] Key type ._llcrypt registered [14747.750962] Key type .llcrypt registered [14747.912436] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [14749.469461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14749.515562] alg: No test for adler32 (adler32-zlib) [14750.779750] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [14751.110199] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [14752.839245] Key type lgssc registered [14754.232711] Lustre: Echo OBD driver; http://www.lustre.org/ [14758.754522] LNet: Added LNI 192.168.202.146@tcp100 [8/256/0/180] [14759.764081] LNet: Added LNI 192.168.202.146@tcp101 [8/256/0/180] [14760.844356] LNet: Added LNI 192.168.202.146@tcp102 [8/256/0/180] [14761.813253] LNet: Added LNI 192.168.202.146@tcp103 [8/256/0/180] [14762.891792] LNet: Added LNI 192.168.202.146@tcp104 [8/256/0/180] [14763.963163] LNet: Added LNI 192.168.202.146@tcp105 [8/256/0/180] [14764.872953] LNet: Added LNI 192.168.202.146@tcp106 [8/256/0/180] [14765.749428] LNet: Added LNI 192.168.202.146@tcp107 [8/256/0/180] [14766.605658] LNet: Added LNI 192.168.202.146@tcp108 [8/256/0/180] [14767.567761] LNet: Added LNI 192.168.202.146@tcp109 [8/256/0/180] [14768.492374] LNet: Added LNI 192.168.202.146@tcp110 [8/256/0/180] [14769.361868] LNet: Added LNI 192.168.202.146@tcp111 [8/256/0/180] [14770.320315] LNet: Added LNI 192.168.202.146@tcp112 [8/256/0/180] [14771.368464] LNet: Added LNI 192.168.202.146@tcp113 [8/256/0/180] [14772.328645] LNet: Added LNI 192.168.202.146@tcp114 [8/256/0/180] [14773.182708] LNet: Added LNI 192.168.202.146@tcp115 [8/256/0/180] [14774.246426] LNet: Added LNI 192.168.202.146@tcp116 [8/256/0/180] [14775.449912] LNet: Added LNI 192.168.202.146@tcp117 [8/256/0/180] [14776.323237] LNet: Added LNI 192.168.202.146@tcp118 [8/256/0/180] [14777.358270] LNet: Added LNI 192.168.202.146@tcp119 [8/256/0/180] [14778.368086] LNet: Added LNI 192.168.202.146@tcp120 [8/256/0/180] [14779.411297] LNet: Added LNI 192.168.202.146@tcp121 [8/256/0/180] [14780.160489] LNet: Added LNI 192.168.202.146@tcp122 [8/256/0/180] [14781.069196] LNet: Added LNI 192.168.202.146@tcp123 [8/256/0/180] [14781.810865] LNet: Added LNI 192.168.202.146@tcp124 [8/256/0/180] [14782.568896] LNet: Added LNI 192.168.202.146@tcp125 [8/256/0/180] [14783.436419] LNet: Added LNI 192.168.202.146@tcp126 [8/256/0/180] [14784.373415] LNet: Added LNI 192.168.202.146@tcp127 [8/256/0/180] [14785.297451] LNet: Added LNI 192.168.202.146@tcp128 [8/256/0/180] [14786.191045] LNet: Added LNI 192.168.202.146@tcp129 [8/256/0/180] [14786.987239] LNet: Added LNI 192.168.202.146@tcp130 [8/256/0/180] [14787.865557] LNet: Added LNI 192.168.202.146@tcp131 [8/256/0/180] [14788.768611] LNet: Added LNI 192.168.202.146@tcp132 [8/256/0/180] [14789.638974] LNet: Added LNI 192.168.202.146@tcp133 [8/256/0/180] [14790.610531] LNet: Added LNI 192.168.202.146@tcp134 [8/256/0/180] [14791.556613] LNet: Added LNI 192.168.202.146@tcp135 [8/256/0/180] [14798.697863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14798.738751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14799.957986] Lustre: 464401:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.146@tcp: can't add NID 192.168.202.146@tcp130: rc = -75 [14800.095979] LNetError: 464423:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.146 [14800.109195] LNetError: Protocol error connecting to 192.168.202.146@tcp122 at host 192.168.202.146:988: is it running a compatible version of Lustre? [14800.188955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14800.225768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14800.287186] LNet: 8 local NIs in recovery (showing 8): 192.168.202.146@tcp122, 192.168.202.146@tcp124, 192.168.202.146@tcp126, 192.168.202.146@tcp127, 192.168.202.146@tcp123, 192.168.202.146@tcp129, 192.168.202.146@tcp128, 192.168.202.146@tcp130 [14800.329206] Lustre: lustre-MDT0000: new disk, initializing [14800.424229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14800.449136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14801.184302] LNetError: 464422:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.146 [14801.198938] LNetError: 464422:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 14 previous similar messages [14801.204387] LNetError: Protocol error connecting to 192.168.202.146@tcp108 at host 192.168.202.146:988: is it running a compatible version of Lustre? [14801.213486] LNetError: Skipped 14 previous similar messages [14803.360505] LNetError: 464427:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.146 [14803.373975] LNetError: 464427:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [14803.395414] LNetError: Protocol error connecting to 192.168.202.146@tcp123 at host 192.168.202.146:988: is it running a compatible version of Lustre? [14803.409807] LNetError: Skipped 10 previous similar messages [14805.628318] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [14807.460281] LNetError: 461858:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.146 [14807.475767] LNetError: Protocol error connecting to 192.168.202.146@tcp123 at host 192.168.202.146:988: is it running a compatible version of Lustre? [14815.650976] LNetError: 464421:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.146 [14815.665497] LNetError: Protocol error connecting to 192.168.202.146@tcp123 at host 192.168.202.146:988: is it running a compatible version of Lustre? [14816.464485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14816.619413] Lustre: 464416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14816.691734] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14816.696482] Lustre: Skipped 1 previous similar message [14816.775433] Lustre: lustre-MDT0001: new disk, initializing [14816.830554] Lustre: 465222:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.146@tcp: can't add NID 192.168.202.146@tcp130: rc = -75 [14816.842401] Lustre: 465222:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14816.961807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14817.003288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14817.036233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14818.961298] Lustre: 465311:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.146@tcp: can't add NID 192.168.202.146@tcp130: rc = -75 [14818.979897] Lustre: 465311:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14822.630056] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [14823.840676] LNetError: 461858:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.146 [14823.871247] LNetError: 461858:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [14823.896937] LNetError: Protocol error connecting to 192.168.202.146@tcp133 at host 192.168.202.146:988: is it running a compatible version of Lustre? [14823.922269] LNetError: Skipped 10 previous similar messages [14833.805379] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14840.231899] LNetError: 464422:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.146 [14840.259617] LNetError: 464422:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [14840.289500] LNetError: Protocol error connecting to 192.168.202.146@tcp120 at host 192.168.202.146:988: is it running a compatible version of Lustre? [14840.317623] LNetError: Skipped 9 previous similar messages [14840.360293] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14848.215543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14848.733672] Lustre: lustre-OST0000: new disk, initializing [14848.736786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14848.746283] Lustre: 466394:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14848.824154] Lustre: 466399:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.146@tcp: can't add NID 192.168.202.146@tcp130: rc = -75 [14848.829506] Lustre: 466399:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14848.892495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14855.777155] Lustre: 466709:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.146@tcp: can't add NID 192.168.202.146@tcp130: rc = -75 [14855.787495] Lustre: 466709:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14855.913780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14855.932470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14856.199997] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14856.353622] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [14859.682629] LNet: 2 local NIs in recovery (showing 2): 192.168.202.146@tcp133, 192.168.202.146@tcp123 [14867.593803] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14872.997497] LNetError: 464422:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.146 [14873.013207] LNetError: 464422:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 20 previous similar messages [14873.020605] LNetError: Protocol error connecting to 192.168.202.146@tcp124 at host 192.168.202.146:988: is it running a compatible version of Lustre? [14873.032834] LNetError: Skipped 20 previous similar messages [14876.640078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14876.643535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14876.658837] LustreError: Skipped 1 previous similar message [14876.663466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14876.676867] Lustre: Skipped 1 previous similar message [14876.712146] Lustre: Skipped 1 previous similar message [14881.768479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14881.777062] Lustre: Skipped 1 previous similar message [14882.856955] Lustre: server umount lustre-OST0000 complete [14886.890203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14886.907311] Lustre: Skipped 1 previous similar message [14886.913285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14886.929018] Lustre: Skipped 1 previous similar message [14892.001147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14893.209238] Lustre: server umount lustre-MDT0000 complete [14897.217478] LustreError: 464411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786656319 with bad export cookie 634778224403853843 [14897.224309] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14897.229581] LustreError: 464411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14897.770798] Lustre: server umount lustre-MDT0001 complete [14911.069769] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [14914.746476] Key type lgssc unregistered [14915.190557] LNet: 468311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14915.197252] LNetError: 468311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14916.256873] LNet: Removed LNI 192.168.202.146@tcp [14916.268521] LNetError: 468311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14916.279839] LNet: Removed LNI 192.168.202.146@tcp100 [14917.343325] LNet: Removed LNI 192.168.202.146@tcp101 [14917.345252] LNetError: 468311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14917.352890] LNetError: 468311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14917.367889] LNet: Removed LNI 192.168.202.146@tcp102 [14918.431400] LNet: Removed LNI 192.168.202.146@tcp103 [14919.455689] LNet: Removed LNI 192.168.202.146@tcp104 [14919.461328] LNetError: 468311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14919.472599] LNetError: 468311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14919.479656] LNet: Removed LNI 192.168.202.146@tcp105 [14920.543812] LNet: Removed LNI 192.168.202.146@tcp106 [14921.569034] LNet: Removed LNI 192.168.202.146@tcp107 [14922.655316] LNet: Removed LNI 192.168.202.146@tcp108 [14923.679467] LNet: Removed LNI 192.168.202.146@tcp109 [14923.681895] LNetError: 468311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14923.705531] LNetError: 468311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14923.713897] LNet: Removed LNI 192.168.202.146@tcp110 [14924.767257] LNet: Removed LNI 192.168.202.146@tcp111 [14925.791227] LNet: Removed LNI 192.168.202.146@tcp112 [14926.815349] LNet: Removed LNI 192.168.202.146@tcp113 [14927.839997] LNet: Removed LNI 192.168.202.146@tcp114 [14928.864567] LNet: Removed LNI 192.168.202.146@tcp115 [14929.887605] LNet: Removed LNI 192.168.202.146@tcp116 [14930.911296] LNet: Removed LNI 192.168.202.146@tcp117 [14931.999384] LNet: Removed LNI 192.168.202.146@tcp118 [14932.008971] LNetError: 468311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14932.014927] LNetError: 468311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14932.030441] LNet: Removed LNI 192.168.202.146@tcp119 [14933.093147] LNet: Removed LNI 192.168.202.146@tcp120 [14933.104519] LNet: Removed LNI 192.168.202.146@tcp121 [14934.175358] LNet: Removed LNI 192.168.202.146@tcp122 [14934.186215] LNet: Removed LNI 192.168.202.146@tcp123 [14935.199661] LNet: Removed LNI 192.168.202.146@tcp124 [14936.223394] LNet: Removed LNI 192.168.202.146@tcp125 [14937.247182] LNet: Removed LNI 192.168.202.146@tcp126 [14938.271743] LNet: Removed LNI 192.168.202.146@tcp127 [14939.295167] LNet: Removed LNI 192.168.202.146@tcp128 [14940.319214] LNet: Removed LNI 192.168.202.146@tcp129 [14941.343489] LNet: Removed LNI 192.168.202.146@tcp130 [14942.367642] LNet: Removed LNI 192.168.202.146@tcp131 [14943.391464] LNet: Removed LNI 192.168.202.146@tcp132 [14943.393569] LNet: Removed LNI 192.168.202.146@tcp133 [14944.415168] LNet: Removed LNI 192.168.202.146@tcp134 [14945.448059] LNet: Removed LNI 192.168.202.146@tcp135 [14946.239346] Key type .llcrypt unregistered [14946.241424] Key type ._llcrypt unregistered [14973.126950] Key type ._llcrypt registered [14973.129382] Key type .llcrypt registered [14973.247575] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [14990.330992] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [14991.842347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14991.909104] alg: No test for adler32 (adler32-zlib) [14993.036901] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [14993.343910] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [14995.103262] Key type lgssc registered [14996.295503] Lustre: Echo OBD driver; http://www.lustre.org/ [15032.655365] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 17:27:32 (1786656452) [15051.199912] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [15053.896982] Key type lgssc unregistered [15054.172665] LNet: 472404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15054.176841] LNetError: 472404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15054.224578] LNet: Removed LNI 192.168.202.146@tcp [15055.040473] Key type .llcrypt unregistered [15055.047962] Key type ._llcrypt unregistered [15072.852891] Key type ._llcrypt registered [15072.860023] Key type .llcrypt registered [15072.971382] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [15074.194203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15074.225126] alg: No test for adler32 (adler32-zlib) [15075.305877] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [15075.669832] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [15077.320801] Key type lgssc registered [15078.978882] Lustre: Echo OBD driver; http://www.lustre.org/ [15090.290932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15090.312597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15091.850247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15091.938178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15092.066714] Lustre: lustre-MDT0000: new disk, initializing [15092.123084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15092.148795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15096.601223] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [15107.266828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15107.490965] Lustre: 473620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15107.562827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15107.566666] Lustre: Skipped 1 previous similar message [15107.687122] Lustre: lustre-MDT0001: new disk, initializing [15107.832363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15107.872983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15107.883447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15113.417885] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [15122.923192] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15128.215488] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15137.026250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15137.575501] Lustre: lustre-OST0000: new disk, initializing [15137.577976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15137.591908] Lustre: 475587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15137.725870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15145.551914] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [15146.026746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15146.042777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15146.188182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15155.397389] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15159.915539] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing lnet_if_list [15166.355283] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15166.399974] LNet: Added LNI 192.168.202.146@tcp10 [8/256/0/180] [15174.313663] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15174.392612] LNet: Added LNI 192.168.202.146@tcp11 [8/256/0/180] [15180.452909] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15180.482578] LNet: Added LNI 192.168.202.146@tcp12 [8/256/0/180] [15185.886467] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15185.937300] LNet: Added LNI 192.168.202.146@tcp13 [8/256/0/180] [15192.087212] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15192.143927] LNet: Added LNI 192.168.202.146@tcp14 [8/256/0/180] [15198.114065] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15198.146885] LNet: Added LNI 192.168.202.146@tcp15 [8/256/0/180] [15204.241424] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15204.264435] LNet: Added LNI 192.168.202.146@tcp16 [8/256/0/180] [15209.634552] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15209.651381] LNet: Added LNI 192.168.202.146@tcp17 [8/256/0/180] [15216.373792] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15216.400588] LNet: Added LNI 192.168.202.146@tcp18 [8/256/0/180] [15222.806838] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15222.871832] LNet: Added LNI 192.168.202.146@tcp19 [8/256/0/180] [15230.572167] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15230.597276] LNet: Added LNI 192.168.202.146@tcp20 [8/256/0/180] [15237.927285] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15237.948354] LNet: Added LNI 192.168.202.146@tcp21 [8/256/0/180] [15243.663382] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15243.705589] LNet: Added LNI 192.168.202.146@tcp22 [8/256/0/180] [15249.678301] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15249.711669] LNet: Added LNI 192.168.202.146@tcp23 [8/256/0/180] [15255.775779] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15255.794578] LNet: Added LNI 192.168.202.146@tcp24 [8/256/0/180] [15261.801165] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15261.854048] LNet: Added LNI 192.168.202.146@tcp25 [8/256/0/180] [15269.054979] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15269.160159] LNet: Added LNI 192.168.202.146@tcp26 [8/256/0/180] [15275.865538] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15275.915579] LNet: Added LNI 192.168.202.146@tcp27 [8/256/0/180] [15281.654460] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15281.695771] LNet: Added LNI 192.168.202.146@tcp28 [8/256/0/180] [15290.276872] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15290.320388] LNet: Added LNI 192.168.202.146@tcp29 [8/256/0/180] [15298.137219] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15298.159562] LNet: Added LNI 192.168.202.146@tcp30 [8/256/0/180] [15305.244285] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15305.276131] LNet: Added LNI 192.168.202.146@tcp31 [8/256/0/180] [15313.146578] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15313.188378] LNet: Added LNI 192.168.202.146@tcp32 [8/256/0/180] [15321.839357] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15321.871971] LNet: Added LNI 192.168.202.146@tcp33 [8/256/0/180] [15331.201655] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15331.246906] LNet: Added LNI 192.168.202.146@tcp34 [8/256/0/180] [15340.190500] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15340.216591] LNet: Added LNI 192.168.202.146@tcp35 [8/256/0/180] [15349.080932] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15349.113222] LNet: Added LNI 192.168.202.146@tcp36 [8/256/0/180] [15358.187148] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15358.245348] LNet: Added LNI 192.168.202.146@tcp37 [8/256/0/180] [15365.070859] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15365.093987] LNet: Added LNI 192.168.202.146@tcp38 [8/256/0/180] [15373.013923] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15373.048675] LNet: Added LNI 192.168.202.146@tcp39 [8/256/0/180] [15380.992704] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15381.014938] LNet: Added LNI 192.168.202.146@tcp40 [8/256/0/180] [15390.591476] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15390.621830] LNet: Added LNI 192.168.202.146@tcp41 [8/256/0/180] [15400.906731] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15400.933399] LNet: Added LNI 192.168.202.146@tcp42 [8/256/0/180] [15410.488813] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15410.544787] LNet: Added LNI 192.168.202.146@tcp43 [8/256/0/180] [15420.468953] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15420.508350] LNet: Added LNI 192.168.202.146@tcp44 [8/256/0/180] [15430.720398] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15430.745781] LNet: Added LNI 192.168.202.146@tcp45 [8/256/0/180] [15441.440527] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15441.482439] LNet: Added LNI 192.168.202.146@tcp46 [8/256/0/180] [15452.027035] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15452.089549] LNet: Added LNI 192.168.202.146@tcp47 [8/256/0/180] [15464.940526] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15464.964317] LNet: Added LNI 192.168.202.146@tcp48 [8/256/0/180] [15476.237966] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15476.272707] LNet: Added LNI 192.168.202.146@tcp49 [8/256/0/180] [15486.308700] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15486.365242] LNet: Added LNI 192.168.202.146@tcp50 [8/256/0/180] [15509.999871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15510.014295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15510.026086] Lustre: Skipped 2 previous similar messages [15510.038382] Lustre: Skipped 2 previous similar messages [15513.892368] Lustre: server umount lustre-MDT0000 complete [15518.111158] LustreError: 473613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786656940 with bad export cookie 6694945960723469667 [15518.123835] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15518.814817] Lustre: server umount lustre-MDT0001 complete [15535.519095] Lustre: 472893:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786656942/real 1786656942] req@000000008eb81d30 x1873445124236160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786656958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15535.548406] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15538.656428] Lustre: 472890:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786656945/real 1786656945] req@0000000047727a7e x1873445124236288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786656961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15539.192333] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [15540.717169] Lustre: 472891:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786656947/real 1786656947] req@00000000ac2e60fe x1873445124236416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786656963 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15543.711231] Lustre: 472893:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786656950/real 1786656950] req@00000000b06023cc x1873445124236672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786656966 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15549.694383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15549.919426] Lustre: 472892:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786656956/real 1786656956] req@000000005b965bde x1873445124236800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786656972 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15563.662612] LustreError: 475581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15563.686423] LustreError: 475581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15563.730772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15564.769710] LustreError: 475582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15564.778665] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15566.821764] LustreError: 475581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15568.520919] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [15571.943597] LustreError: 475582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15577.061587] LustreError: 475581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15577.254732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15577.489208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15580.593955] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15581.839881] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [15589.813469] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15595.738168] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15618.532121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15618.546278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15618.573574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15620.064633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15620.078658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15626.727573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15626.746656] Lustre: Skipped 3 previous similar messages [15631.327338] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15631.697575] Lustre: server umount lustre-OST0000 complete [15636.961620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15636.982882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15636.986993] Lustre: Skipped 2 previous similar messages [15637.003700] Lustre: Skipped 1 previous similar message [15642.708951] Lustre: server umount lustre-MDT0000 complete [15647.130195] LustreError: 473613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657069 with bad export cookie 6694945960723472215 [15647.140027] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15647.146802] LustreError: 473613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15647.452081] Lustre: server umount lustre-MDT0001 complete [15659.859961] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [15662.957891] Key type lgssc unregistered [15663.317596] LNet: 502639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15663.328754] LNetError: 502639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15664.351152] LNet: Removed LNI 192.168.202.146@tcp [15664.353824] LNetError: 502639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15664.362363] LNet: Removed LNI 192.168.202.146@tcp10 [15664.366437] LNet: Removed LNI 192.168.202.146@tcp11 [15664.371883] LNet: Removed LNI 192.168.202.146@tcp12 [15664.376525] LNet: Removed LNI 192.168.202.146@tcp13 [15664.379775] LNet: Removed LNI 192.168.202.146@tcp14 [15664.388954] LNet: Removed LNI 192.168.202.146@tcp15 [15664.400372] LNet: Removed LNI 192.168.202.146@tcp16 [15664.402152] LNet: Removed LNI 192.168.202.146@tcp17 [15664.405724] LNet: Removed LNI 192.168.202.146@tcp18 [15664.410708] LNet: Removed LNI 192.168.202.146@tcp19 [15664.415843] LNet: Removed LNI 192.168.202.146@tcp20 [15664.429411] LNet: Removed LNI 192.168.202.146@tcp21 [15664.434859] LNet: Removed LNI 192.168.202.146@tcp22 [15664.441567] LNet: Removed LNI 192.168.202.146@tcp23 [15664.447484] LNet: Removed LNI 192.168.202.146@tcp24 [15664.456642] LNet: Removed LNI 192.168.202.146@tcp25 [15664.462990] LNet: Removed LNI 192.168.202.146@tcp26 [15664.469109] LNet: Removed LNI 192.168.202.146@tcp27 [15664.475310] LNet: Removed LNI 192.168.202.146@tcp28 [15664.478664] LNet: Removed LNI 192.168.202.146@tcp29 [15664.481658] LNet: Removed LNI 192.168.202.146@tcp30 [15664.484460] LNet: Removed LNI 192.168.202.146@tcp31 [15664.487736] LNet: Removed LNI 192.168.202.146@tcp32 [15664.490707] LNet: Removed LNI 192.168.202.146@tcp33 [15664.493561] LNet: Removed LNI 192.168.202.146@tcp34 [15664.496605] LNet: Removed LNI 192.168.202.146@tcp35 [15664.498682] LNet: Removed LNI 192.168.202.146@tcp36 [15664.500741] LNet: Removed LNI 192.168.202.146@tcp37 [15664.503125] LNet: Removed LNI 192.168.202.146@tcp38 [15664.506458] LNet: Removed LNI 192.168.202.146@tcp39 [15664.509363] LNet: Removed LNI 192.168.202.146@tcp40 [15664.512159] LNet: Removed LNI 192.168.202.146@tcp41 [15664.514044] LNet: Removed LNI 192.168.202.146@tcp42 [15664.516394] LNet: Removed LNI 192.168.202.146@tcp43 [15664.519852] LNet: Removed LNI 192.168.202.146@tcp44 [15664.523127] LNet: Removed LNI 192.168.202.146@tcp45 [15664.525963] LNet: Removed LNI 192.168.202.146@tcp46 [15664.527854] LNet: Removed LNI 192.168.202.146@tcp47 [15664.529781] LNet: Removed LNI 192.168.202.146@tcp48 [15664.532078] LNet: Removed LNI 192.168.202.146@tcp49 [15664.544990] LNet: Removed LNI 192.168.202.146@tcp50 [15665.378226] Key type .llcrypt unregistered [15665.382432] Key type ._llcrypt unregistered [15686.295199] Key type ._llcrypt registered [15686.298095] Key type .llcrypt registered [15686.414229] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [15703.406849] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [15704.719416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15704.803971] alg: No test for adler32 (adler32-zlib) [15705.993594] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [15706.333890] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [15708.071559] Key type lgssc registered [15709.333604] Lustre: Echo OBD driver; http://www.lustre.org/ [15747.645551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15747.674363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15748.953344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15748.992581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15749.064752] Lustre: lustre-MDT0000: new disk, initializing [15749.121348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15749.135420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15753.967240] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [15765.715348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15765.945924] Lustre: 505843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15766.045533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15766.051531] Lustre: Skipped 1 previous similar message [15766.126341] Lustre: lustre-MDT0001: new disk, initializing [15766.313600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15766.379786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15766.391882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15770.956956] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [15780.875543] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15787.236893] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15795.877282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15796.092266] Lustre: *** cfs_fail_loc=90f, val=0*** [15796.170299] Lustre: lustre-OST0000: new disk, initializing [15796.174499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15796.187397] Lustre: 507858:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15796.289938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15803.205281] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [15806.038885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15806.052234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15806.243102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15813.814349] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15821.804829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15821.813705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15821.819646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15821.854234] Lustre: Skipped 1 previous similar message [15827.211644] Lustre: server umount lustre-OST0000 complete [15832.037528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15832.046507] Lustre: Skipped 1 previous similar message [15832.050395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15832.062331] Lustre: Skipped 1 previous similar message [15837.155755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15837.171893] Lustre: Skipped 1 previous similar message [15842.273591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15842.277882] Lustre: Skipped 1 previous similar message [15845.343138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15845.623922] Lustre: server umount lustre-MDT0000 complete [15847.399813] LustreError: 505854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15847.428404] LustreError: 505854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15849.144759] LustreError: 505833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657271 with bad export cookie 7278950215316085094 [15849.167301] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15849.678514] Lustre: server umount lustre-MDT0001 complete [15860.265229] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [15863.573132] Key type lgssc unregistered [15863.851550] LNet: 509768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15863.855500] LNetError: 509768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15863.885233] LNet: Removed LNI 192.168.202.146@tcp [15864.752267] Key type .llcrypt unregistered [15864.764361] Key type ._llcrypt unregistered [15886.209548] Key type ._llcrypt registered [15886.213563] Key type .llcrypt registered [15886.341236] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [15887.724301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15887.857616] alg: No test for adler32 (adler32-zlib) [15888.967891] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [15889.192127] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [15890.863302] Key type lgssc registered [15892.123236] Lustre: Echo OBD driver; http://www.lustre.org/ [15899.427138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15899.462264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15900.926455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15905.091359] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [15911.406899] LustreError: 511040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15911.430258] LustreError: 511040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15914.088238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15914.311785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15918.330258] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [15927.112222] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15932.851451] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15940.621274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15941.127298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15949.093515] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [15957.792740] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15969.827856] loop0: detected capacity change from 0 to 204800000 [15970.019963] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15970.032887] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15970.058243] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15977.203243] loop0: detected capacity change from 0 to 204800000 [15977.343348] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15977.368392] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15977.412019] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15982.918447] loop0: detected capacity change from 0 to 204800000 [15989.169736] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15989.371780] Lustre: fs15246-OST0000: new disk, initializing [15989.376690] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15989.383179] Lustre: 514636:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15989.466495] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15995.038528] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [16005.057883] loop1: detected capacity change from 0 to 204800000 [16012.231358] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16012.483244] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16012.597642] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16012.802490] Lustre: fs15246-MDT0000: new disk, initializing [16013.147338] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16013.191232] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16013.418438] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16014.499580] Lustre: 515759:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000043173be0 x1873445976588544/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786657453 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16018.187534] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [16023.662519] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16023.690377] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16023.742830] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16041.963759] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16042.003575] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16047.129640] LustreError: 511036:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16047.212160] Lustre: server umount fs15246-MDT0000 complete [16052.218267] LustreError: 513093:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16056.764485] Lustre: server umount fs15246-OST0000 complete [16072.682032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16072.685839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16072.702259] Lustre: Skipped 1 previous similar message [16072.727648] Lustre: Skipped 1 previous similar message [16077.812050] LustreError: 513530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16078.118714] Lustre: server umount lustre-OST0000 complete [16083.426713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16083.437955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16083.454799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16088.033919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16088.035491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16088.074182] Lustre: Skipped 1 previous similar message [16089.372499] Lustre: server umount lustre-MDT0000 complete [16093.047461] LustreError: 511019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657515 with bad export cookie 7685425849879878128 [16093.051336] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16093.056581] LustreError: 511019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16093.665427] Lustre: server umount lustre-MDT0001 complete [16105.099909] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [16108.763109] Key type lgssc unregistered [16109.172037] LNet: 518864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16109.185206] LNetError: 518864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16109.217696] LNet: Removed LNI 192.168.202.146@tcp [16110.173045] Key type .llcrypt unregistered [16110.184350] Key type ._llcrypt unregistered [16130.424648] Key type ._llcrypt registered [16130.426527] Key type .llcrypt registered [16130.506314] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [16131.573880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16131.719701] alg: No test for adler32 (adler32-zlib) [16132.957195] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [16133.278507] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [16135.039221] Key type lgssc registered [16136.482942] Lustre: Echo OBD driver; http://www.lustre.org/ [16180.735774] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [16193.050585] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [16230.491568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16230.537733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16232.005563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16232.107226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16232.214547] Lustre: lustre-MDT0000: new disk, initializing [16232.327841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16232.357234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16236.299495] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [16245.948029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16246.064946] Lustre: 523335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16246.123783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16246.132758] Lustre: Skipped 1 previous similar message [16246.211691] Lustre: lustre-MDT0001: new disk, initializing [16246.344955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16246.391903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16246.408383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16250.492739] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [16261.723247] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16267.307079] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16274.249130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16274.564495] Lustre: lustre-OST0000: new disk, initializing [16274.567740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16274.574719] Lustre: 525302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16274.719649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16281.163044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16281.172302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16281.293346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16281.713941] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [16291.670109] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16297.092390] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16297.354514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16303.349237] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16303.644980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16307.169946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16307.185628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16307.188699] LustreError: Skipped 1 previous similar message [16307.191231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16307.244262] Lustre: Skipped 1 previous similar message [16311.948816] Lustre: server umount lustre-OST0000 complete [16317.411344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16317.418893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16317.433201] Lustre: Skipped 1 previous similar message [16317.458759] Lustre: Skipped 2 previous similar messages [16322.337784] Lustre: server umount lustre-MDT0000 complete [16326.340794] LustreError: 523327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657748 with bad export cookie 1570445160801841529 [16326.346698] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16326.352516] LustreError: 523327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16326.661711] Lustre: server umount lustre-MDT0001 complete [16352.616803] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [16362.957276] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [16396.082974] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16093 sec ========= 17:50:15 (1786657815) [16398.189612] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:50:18 (1786657818) === [16402.222616] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:50:22 (1786657822) === [16446.635099] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [16450.518480] Key type lgssc unregistered [16450.754188] LNet: 531952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16450.761670] LNetError: 531952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16450.780716] LNet: Removed LNI 192.168.202.146@tcp [16451.652530] Key type .llcrypt unregistered [16451.654665] Key type ._llcrypt unregistered