[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509977506 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003273] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008402] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011117] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014067] *** VALIDATE selinux *** [ 0.022504] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027380] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029112] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032036] *** VALIDATE tmpfs *** [ 0.033444] *** VALIDATE proc *** [ 0.035201] *** VALIDATE cgroup *** [ 0.036010] *** VALIDATE cgroup2 *** [ 0.037207] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038118] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041025] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.045009] debug: unmapping init [mem 0xffffffffb6859000-0xffffffffb6860fff] [ 0.047921] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048622] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049021] ... version: 2 [ 0.049939] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.050864] ... value mask: 0000ffffffffffff [ 0.051012] ... max period: 00007fffffffffff [ 0.052009] ... fixed-purpose events: 3 [ 0.052944] ... event mask: 000000070000000f [ 0.053267] rcu: Hierarchical SRCU implementation. [ 0.055310] smp: Bringing up secondary CPUs ... [ 0.056568] x86: Booting SMP configuration: [ 0.057019] .... node #0, CPUs: #1 #2 #3 [ 0.061750] smp: Brought up 1 node, 4 CPUs [ 0.063020] smpboot: Max logical packages: 1 [ 0.064011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.126653] node 0 deferred pages initialised in 60ms [ 0.128195] devtmpfs: initialized [ 0.129309] x86/mm: Memory block size: 128MB [ 0.133758] gcov: version magic: 0x41383552 [ 0.136341] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139194] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141430] pinctrl core: initialized pinctrl subsystem [ 0.143226] [ 0.143938] ************************************************************* [ 0.147019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149009] ** ** [ 0.150010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153018] ** ** [ 0.155016] ** This means that this kernel is built to expose internal ** [ 0.157029] ** IOMMU data structures, which may compromise security on ** [ 0.158007] ** your system. ** [ 0.160010] ** ** [ 0.162010] ** If you see this message and you are not debugging the ** [ 0.164009] ** kernel, report this immediately to your vendor! ** [ 0.165009] ** ** [ 0.167013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168009] ************************************************************* [ 0.170691] NET: Registered protocol family 16 [ 0.172487] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179021] cpuidle: using governor menu [ 0.180472] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182525] PCI: Using configuration type 1 for base access [ 0.185173] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195077] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.201046] cryptd: max_cpu_qlen set to 1000 [ 0.204354] ACPI: Added _OSI(Module Device) [ 0.206013] ACPI: Added _OSI(Processor Device) [ 0.207012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.209013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214522] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.222325] ACPI: Interpreter enabled [ 0.223075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.224012] ACPI: Using IOAPIC for interrupt routing [ 0.225104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.226499] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234449] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240035] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244170] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.250365] acpiphp: Slot [2] registered [ 0.252130] acpiphp: Slot [5] registered [ 0.253160] acpiphp: Slot [6] registered [ 0.255113] acpiphp: Slot [7] registered [ 0.257108] acpiphp: Slot [8] registered [ 0.258118] acpiphp: Slot [9] registered [ 0.260091] acpiphp: Slot [10] registered [ 0.261122] acpiphp: Slot [3] registered [ 0.263099] acpiphp: Slot [4] registered [ 0.265112] acpiphp: Slot [11] registered [ 0.266111] acpiphp: Slot [12] registered [ 0.268190] acpiphp: Slot [13] registered [ 0.269105] acpiphp: Slot [14] registered [ 0.271096] acpiphp: Slot [15] registered [ 0.273174] acpiphp: Slot [16] registered [ 0.274097] acpiphp: Slot [17] registered [ 0.276148] acpiphp: Slot [18] registered [ 0.277133] acpiphp: Slot [19] registered [ 0.279157] acpiphp: Slot [20] registered [ 0.280153] acpiphp: Slot [21] registered [ 0.282078] acpiphp: Slot [22] registered [ 0.282942] acpiphp: Slot [23] registered [ 0.283257] acpiphp: Slot [24] registered [ 0.285170] acpiphp: Slot [25] registered [ 0.287121] acpiphp: Slot [26] registered [ 0.289138] acpiphp: Slot [27] registered [ 0.291082] acpiphp: Slot [28] registered [ 0.292053] acpiphp: Slot [29] registered [ 0.293055] acpiphp: Slot [30] registered [ 0.294095] acpiphp: Slot [31] registered [ 0.295042] PCI host bridge to bus 0000:00 [ 0.296017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.299013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.301015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.303032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.306026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.308174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.311060] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.313655] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.323798] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.329894] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.332020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.335019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.337074] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.340699] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.343871] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.346045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.349756] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.355017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.364014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.367011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.371172] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.378018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.382016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.394016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.404000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.416017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.433026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.454017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.462528] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.467018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.473016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.483017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.495292] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.501014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.507017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.519062] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.532000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.544055] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.552015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.572019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.583000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.588023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.594016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.603014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.610973] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.613234] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.615332] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.616241] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.618138] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.622006] iommu: Default domain type: Passthrough [ 0.623214] SCSI subsystem initialized [ 0.626167] ACPI: bus type USB registered [ 0.629136] usbcore: registered new interface driver usbfs [ 0.631089] usbcore: registered new interface driver hub [ 0.633081] usbcore: registered new device driver usb [ 0.635184] pps_core: LinuxPPS API ver. 1 registered [ 0.638014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.641079] PTP clock support registered [ 0.642297] EDAC MC: Ver: 3.0.0 [ 0.644161] PCI: Using ACPI for IRQ routing [ 0.645850] NetLabel: Initializing [ 0.647011] NetLabel: domain hash size = 128 [ 0.649012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.651084] NetLabel: unlabeled traffic allowed by default [ 0.653141] vgaarb: loaded [ 0.654424] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.656016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.665363] clocksource: Switched to clocksource kvm-clock [ 0.777311] VFS: Disk quotas dquot_6.6.0 [ 0.778702] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.780863] *** VALIDATE ramfs *** [ 0.782092] *** VALIDATE hugetlbfs *** [ 0.783330] pnp: PnP ACPI init [ 0.785468] pnp: PnP ACPI: found 6 devices [ 0.810798] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.814345] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.816597] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.818742] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.821305] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.824078] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.826296] NET: Registered protocol family 2 [ 0.829186] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.833927] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.837203] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.841861] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.844526] TCP: Hash tables configured (established 65536 bind 65536) [ 0.847103] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.849780] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.851754] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.854456] NET: Registered protocol family 1 [ 0.857175] RPC: Registered named UNIX socket transport module. [ 0.859288] RPC: Registered udp transport module. [ 0.860744] RPC: Registered tcp transport module. [ 0.861896] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.863669] NET: Registered protocol family 44 [ 0.864924] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.866620] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.868281] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.870353] PCI: CLS 0 bytes, default 64 [ 0.871946] Unpacking initramfs... [ 2.280679] debug: unmapping init [mem 0xffff8f33fcc54000-0xffff8f33fffbffff] [ 2.284797] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.287071] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.289979] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.785208] Initialise system trusted keyrings [ 2.786852] Key type blacklist registered [ 2.788315] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.795399] zbud: loaded [ 2.798363] *** VALIDATE nfs *** [ 2.799186] *** VALIDATE nfs4 *** [ 2.800157] pstore: using deflate compression [ 2.803287] Platform Keyring initialized [ 2.907846] NET: Registered protocol family 38 [ 2.910795] Key type asymmetric registered [ 2.912508] Asymmetric key parser 'x509' registered [ 2.914624] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.917879] io scheduler mq-deadline registered [ 2.919039] io scheduler kyber registered [ 2.920716] io scheduler bfq registered [ 2.922798] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.925541] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.927891] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.930045] ACPI: Power Button [PWRF] [ 2.933845] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.938680] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.946287] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.950815] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.963161] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.989023] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.015889] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.020753] Non-volatile memory driver v1.3 [ 3.021760] Linux agpgart interface v0.103 [ 3.047959] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.049690] vda: detected capacity change from 0 to 76677120 [ 3.061792] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.064662] vdb: detected capacity change from 0 to 1073741824 [ 3.085840] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.088055] vdc: detected capacity change from 0 to 2621440000 [ 3.112320] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.115437] vdd: detected capacity change from 0 to 2621440000 [ 3.140946] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.144281] vde: detected capacity change from 0 to 4294967296 [ 3.164347] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.167267] vdf: detected capacity change from 0 to 4294967296 [ 3.175926] libphy: Fixed MDIO Bus: probed [ 3.182206] usbcore: registered new interface driver usbserial_generic [ 3.184068] usbserial: USB Serial support registered for generic [ 3.186717] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.190079] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.191610] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.193232] mousedev: PS/2 mouse device common for all mice [ 3.195298] rtc_cmos 00:05: RTC can wake from S4 [ 3.198033] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.199208] rtc_cmos 00:05: registered as rtc0 [ 3.204537] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.205912] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.206725] intel_pstate: CPU model not supported [ 3.212073] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.216264] hid: raw HID events driver (C) Jiri Kosina [ 3.218793] usbcore: registered new interface driver usbhid [ 3.221164] usbhid: USB HID core driver [ 3.222451] drop_monitor: Initializing network drop monitor service [ 3.224707] Initializing XFRM netlink socket [ 3.226479] NET: Registered protocol family 10 [ 3.229341] Segment Routing with IPv6 [ 3.230086] NET: Registered protocol family 17 [ 3.231961] mpls_gso: MPLS GSO support [ 3.236887] RAS: Correctable Errors collector initialized. [ 3.238218] AVX version of gcm_enc/dec engaged. [ 3.239065] AES CTR mode by8 optimization enabled [ 3.300538] sched_clock: Marking stable (3300520238, 0)->(4149856443, -849336205) [ 3.302734] registered taskstats version 1 [ 3.303872] Loading compiled-in X.509 certificates [ 3.305121] zswap: loaded using pool lzo/zbud [ 3.324622] Key type big_key registered [ 3.334195] Key type encrypted registered [ 3.335290] ima: No TPM chip found, activating TPM-bypass! [ 3.336604] ima: Allocated hash algorithm: sha1 [ 3.337890] ima: No architecture policies found [ 3.339794] evm: Initialising EVM extended attributes: [ 3.341769] evm: security.selinux [ 3.342988] evm: security.ima [ 3.344206] evm: security.capability [ 3.345589] evm: HMAC attrs: 0x1 [ 3.347928] rtc_cmos 00:05: setting system clock to 2026-09-03 11:58:49 UTC (1788436729) [ 3.354145] debug: unmapping init [mem 0xffffffffb7803000-0xffffffffb79fffff] [ 3.357178] debug: unmapping init [mem 0xffffffffb6582000-0xffffffffb6858fff] [ 3.366069] Write protecting the kernel read-only data: 28672k [ 3.369152] debug: unmapping init [mem 0xffffffffb4c03000-0xffffffffb4dfffff] [ 3.371850] debug: unmapping init [mem 0xffffffffb5514000-0xffffffffb55fffff] [ 3.405554] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.413787] systemd[1]: Detected virtualization kvm. [ 3.415641] systemd[1]: Detected architecture x86-64. [ 3.417404] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.440242] systemd[1]: No hostname configured. [ 3.441526] systemd[1]: Set hostname to . [ 3.443345] random: systemd: uninitialized urandom read (16 bytes read) [ 3.445710] systemd[1]: Initializing machine ID from random generator. [ 3.497301] random: ln: uninitialized urandom read (6 bytes read) [ 3.564816] random: systemd: uninitialized urandom read (16 bytes read) [ 3.566616] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.571148] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.575393] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.084411] device-mapper: uevent: version 1.0.3 [ 4.086132] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.742690] virtio_net virtio0 ens2: renamed from eth0 [ 4.753114] random: fast init done [ 4.766817] scsi host0: ata_piix Starting dracut initqueue hook... [ 4.817137] scsi host1: ata_piix [ 4.818701] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.821262] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.085252] dracut-initqueue[593]: RTNETLINK answers: File exists [ 9.850059] random: crng init done [ 9.851325] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.059551] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.114195] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.372778] SELinux: Disabled at runtime. [ 11.432578] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.437829] systemd[1]: Detected virtualization kvm. [ 11.441028] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.922515] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.925274] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.929672] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.933336] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.936801] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.944170] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.948022] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. [ 12.086228] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.386809] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.740347] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.760379] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.893052] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.900987] EDAC sbridge: Ver: 1.1.2 [ 14.187923] Key type dns_resolver registered [ 14.478727] NFS: Registering the id_resolver key type [ 14.480694] Key type id_resolver registered [ 14.482045] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg136-server login: [ 41.326635] libcfs: loading out-of-tree module taints kernel. [ 41.346047] Key type ._llcrypt registered [ 41.347759] Key type .llcrypt registered [ 41.393312] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 49.593825] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 50.145446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.153898] alg: No test for adler32 (adler32-zlib) [ 51.203854] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 51.538875] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 53.159204] Key type lgssc registered [ 53.752110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 65.211555] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 112.842495] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 128.726607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 128.776639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 130.174965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 130.213446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 130.345157] Lustre: lustre-MDT0000: new disk, initializing [ 130.510535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 130.553865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 136.779285] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 151.895853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.048093] Lustre: 6498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 152.138932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 152.145825] Lustre: Skipped 1 previous similar message [ 152.310507] Lustre: lustre-MDT0001: new disk, initializing [ 152.392923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 152.436156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 152.460796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 157.913893] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.920035] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 168.791543] hrtimer: interrupt took 3011450 ns [ 172.117476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 172.389733] Lustre: lustre-OST0000: new disk, initializing [ 172.393365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 172.402640] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 172.458924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.216711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.232883] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 177.349640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 178.318278] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.857249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.953641] Lustre: lustre-OST0001: new disk, initializing [ 194.971211] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 194.981450] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 195.080793] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.153183] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.352642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 205.366637] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 205.438406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 213.340788] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 218.993956] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 225.753207] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing check_logdir /tmp/testlogs/ [ 231.018171] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing yml_node [ 236.986968] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 239.995529] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 242.460467] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 244.346479] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:02:48 EDT 2026 [ 261.692873] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 263.455422] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 280.548712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 280.560922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.580840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 282.087378] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.095216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 282.108128] Lustre: Skipped 2 previous similar messages [ 282.141810] Lustre: Skipped 3 previous similar messages [ 285.624758] Lustre: server umount lustre-MDT0000 complete [ 289.339592] LustreError: 10393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437015 with bad export cookie 16093242459346357010 [ 289.341498] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 289.346587] LustreError: 10393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 292.321590] LustreError: 6505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 292.322664] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.330934] LustreError: 6505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 292.334265] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 292.346492] Lustre: Skipped 1 previous similar message [ 292.358211] Lustre: Skipped 1 previous similar message [ 294.369082] LustreError: 6505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.370558] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 294.388917] LustreError: 6505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 294.408117] Lustre: Skipped 1 previous similar message [ 295.678545] Lustre: server umount lustre-MDT0001 complete [ 299.879418] Lustre: server umount lustre-OST0000 complete [ 303.713997] Lustre: server umount lustre-OST0001 complete [ 309.703123] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 320.405607] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 356.568526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 356.839540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 356.876565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 357.001602] Lustre: lustre-MDT0000: new disk, initializing [ 357.119254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 357.140721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 361.297342] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 371.431369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.539875] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 371.593484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 371.595790] Lustre: Skipped 1 previous similar message [ 371.669497] Lustre: lustre-MDT0001: new disk, initializing [ 371.755048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 371.797147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 371.813627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 375.772868] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 385.736707] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 391.596796] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 399.103522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 399.430155] Lustre: lustre-OST0000: new disk, initializing [ 399.434359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 399.444291] Lustre: 17042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 399.565531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.350520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 401.369635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 401.484297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 406.512744] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 416.400303] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 421.523468] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 421.719298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 427.562657] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 427.779031] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 432.101064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 432.103893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 432.108098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 432.114075] LustreError: Skipped 1 previous similar message [ 432.148943] Lustre: Skipped 1 previous similar message [ 442.337156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 442.342619] Lustre: Skipped 4 previous similar messages [ 443.871206] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 444.132488] Lustre: server umount lustre-OST0000 complete [ 448.481348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 448.491271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.303722] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 462.560990] Lustre: server umount lustre-MDT0000 complete [ 462.828245] LustreError: 15082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.853422] LustreError: 15082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 467.620208] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437193 with bad export cookie 16093242459346358543 [ 467.622776] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 467.629201] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 468.325863] Lustre: server umount lustre-MDT0001 complete [ 473.369654] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:06:37 (1788437197) [ 476.512491] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 478.419792] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:06:43 (1788437203) [ 484.173147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.565224] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.637238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.534360] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 489.966449] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.108589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.169627] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.197320] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 504.574306] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 512.463877] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 518.002125] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 524.058041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 524.272353] LustreError: 21271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.388694] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 538.361782] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 545.248637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 545.249479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.254983] LustreError: Skipped 1 previous similar message [ 545.262255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 545.262265] Lustre: Skipped 5 previous similar messages [ 545.294593] Lustre: Skipped 2 previous similar messages [ 550.541447] Lustre: server umount lustre-OST0000 complete [ 569.312525] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 569.504598] Lustre: server umount lustre-MDT0000 complete [ 570.852893] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.868478] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 573.546420] LustreError: 19435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437299 with bad export cookie 16093242459346359418 [ 573.555894] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.566145] LustreError: 19435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 573.840901] Lustre: server umount lustre-MDT0001 complete [ 586.321093] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 589.367152] Key type lgssc unregistered [ 589.769376] LNet: 22986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.780350] LNetError: 22986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.818101] LNet: Removed LNI 192.168.201.136@tcp [ 591.036172] Key type .llcrypt unregistered [ 591.039272] Key type ._llcrypt unregistered [ 615.646530] Key type ._llcrypt registered [ 615.648397] Key type .llcrypt registered [ 615.769169] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 616.652140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 616.669277] alg: No test for adler32 (adler32-zlib) [ 618.078735] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 618.361678] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 620.143811] Key type lgssc registered [ 621.585623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 630.259728] loop: module loaded [ 630.330671] loop0: detected capacity change from 0 to 204800000 [ 630.641869] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 630.664594] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 630.972910] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 636.865957] loop0: detected capacity change from 0 to 204800000 [ 637.090681] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.102674] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.110860] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 642.371684] loop0: detected capacity change from 0 to 204800000 [ 648.378250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 648.409323] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.766047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 649.809470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 649.870267] Lustre: lustre-MDT0000: new disk, initializing [ 649.974179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.016438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 655.585188] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 665.311406] loop1: detected capacity change from 0 to 204800000 [ 671.660065] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 672.057169] Lustre: lustre-OST07c6: new disk, initializing [ 672.064950] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 672.070738] Lustre: Skipped 1 previous similar message [ 672.082244] Lustre: 26167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 672.195710] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 680.011403] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 681.111375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 681.123757] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 681.177903] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 686.142845] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 686.155600] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 686.160336] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 686.169060] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 686.177365] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 686.193944] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 703.073248] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 703.095446] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 703.099806] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 703.120638] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 703.132911] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 703.138884] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 708.075957] Lustre: 26781:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 708.081977] Lustre: 26781:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 708.086180] Lustre: 26781:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 708.093988] Lustre: 26781:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.100494] Lustre: 26781:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 708.105206] Lustre: 26781:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.108938] Lustre: 26781:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 708.115880] Lustre: 26781:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.120233] Lustre: 26781:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 708.124506] Lustre: 26781:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.128830] Lustre: 26781:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 708.133136] Lustre: 26781:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.282825] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 711.310238] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 711.325485] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 711.334854] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.364328] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 711.372740] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.383761] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 711.390125] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.405960] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 711.425504] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.441946] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 711.459185] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.296645] Lustre: 26781:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 716.313558] Lustre: 26781:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 716.324053] Lustre: 26781:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 716.337763] Lustre: 26781:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.348017] Lustre: 26781:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 716.358521] Lustre: 26781:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.370939] Lustre: 26781:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 716.382646] Lustre: 26781:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.396574] Lustre: 26781:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 716.409284] Lustre: 26781:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.421385] Lustre: 26781:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 716.431865] Lustre: 26781:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.377612] Lustre: 26781:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 728.409702] Lustre: 26781:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 728.432087] Lustre: 26781:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 728.443626] Lustre: 26781:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 728.448169] Lustre: 26781:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 728.452447] Lustre: 26781:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 728.456825] Lustre: 26781:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 728.461195] Lustre: 26781:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 728.465451] Lustre: 26781:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 728.469322] Lustre: 26781:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 728.475244] Lustre: 26781:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 728.486206] Lustre: 26781:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 778.213482] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 778.237077] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 782.723045] Lustre: server umount lustre-OST07c6 complete [ 793.383572] Lustre: server umount lustre-MDT0000 complete [ 810.025165] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 812.845660] Key type lgssc unregistered [ 813.162698] LNet: 28723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.178744] LNetError: 28723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.191110] LNet: Removed LNI 192.168.201.136@tcp [ 813.945674] Key type .llcrypt unregistered [ 813.951270] Key type ._llcrypt unregistered [ 833.766706] Key type ._llcrypt registered [ 833.769301] Key type .llcrypt registered [ 833.857633] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 834.835939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 835.054896] alg: No test for adler32 (adler32-zlib) [ 836.069665] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 836.304502] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 837.999341] Key type lgssc registered [ 839.063372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 847.012683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 847.036117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.596113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 852.975938] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 858.105693] LustreError: 29992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 858.117866] LustreError: 29992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 860.909225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.359583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 865.284474] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 873.894662] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 880.677393] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 889.594192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 890.047742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 893.129657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 896.809417] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 904.951857] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 922.085530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 922.088437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.096829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 922.101118] Lustre: Skipped 1 previous similar message [ 922.113298] Lustre: Skipped 1 previous similar message [ 925.891646] Lustre: server umount lustre-OST0000 complete [ 932.321347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.340365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 936.451527] Lustre: server umount lustre-MDT0000 complete [ 937.448529] LustreError: 29986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.476529] LustreError: 29986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 940.623642] LustreError: 30725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437666 with bad export cookie 11685983339890483521 [ 940.632895] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 940.645888] LustreError: 30725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 941.015209] Lustre: server umount lustre-MDT0001 complete [ 951.433521] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 954.513293] Key type lgssc unregistered [ 954.821085] LNet: 33727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.834857] LNetError: 33727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.851315] LNet: Removed LNI 192.168.201.136@tcp [ 955.727676] Key type .llcrypt unregistered [ 955.730064] Key type ._llcrypt unregistered [ 974.892345] Key type ._llcrypt registered [ 974.895042] Key type .llcrypt registered [ 975.071546] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 976.465504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 976.574039] alg: No test for adler32 (adler32-zlib) [ 977.863513] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 978.223929] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 979.967243] Key type lgssc registered [ 981.332827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 990.224520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 990.260307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 991.633727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.230682] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1000.930805] LustreError: 34997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1000.951211] LustreError: 34997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1004.633943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.936988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1009.248729] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1016.794458] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1022.978208] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1030.110421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1030.591362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.866334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1037.487171] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1045.521303] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1057.249076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1057.258248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.290830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1057.300976] Lustre: Skipped 1 previous similar message [ 1062.373598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1062.384386] Lustre: Skipped 1 previous similar message [ 1062.586980] Lustre: server umount lustre-OST0000 complete [ 1066.477627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1066.494722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1066.512815] Lustre: Skipped 1 previous similar message [ 1066.518839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.610775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.628790] Lustre: Skipped 3 previous similar messages [ 1072.837545] Lustre: server umount lustre-MDT0000 complete [ 1077.678609] LustreError: 34978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437803 with bad export cookie 8496591993797589302 [ 1077.682952] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1077.688937] LustreError: 34978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1078.268784] Lustre: server umount lustre-MDT0001 complete [ 1092.949685] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1096.330594] Key type lgssc unregistered [ 1096.756411] LNet: 38530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.776683] LNetError: 38530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.799856] LNet: Removed LNI 192.168.201.136@tcp [ 1098.255459] Key type .llcrypt unregistered [ 1098.258186] Key type ._llcrypt unregistered [ 1124.610358] Key type ._llcrypt registered [ 1124.616230] Key type .llcrypt registered [ 1124.807918] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1126.414805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1126.531069] alg: No test for adler32 (adler32-zlib) [ 1127.683585] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1127.968459] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1129.679174] Key type lgssc registered [ 1130.928408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1138.963496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1138.994779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.484192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.244518] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1149.951488] LustreError: 39946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.989450] LustreError: 39946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1154.273339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.610583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1159.257676] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1166.424937] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1172.399864] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1178.922669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1179.078956] LustreError: 41769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1179.352336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1185.756154] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1185.787160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1192.993182] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1199.919241] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 1200.608106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1200.617103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.637369] LustreError: 39946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1200.825160] Lustre: server umount lustre-MDT0000 complete [ 1201.133715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.187969] LustreError: 39927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437930 with bad export cookie 11502668834731838155 [ 1204.189352] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1204.200561] LustreError: 39927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1205.007141] LustreError: 39940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.013935] LustreError: 39940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1205.032675] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [ 1206.241184] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.258836] Lustre: Skipped 1 previous similar message [ 1206.262864] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1209.259437] LustreError: 39941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.259726] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1209.274554] LustreError: 39941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1210.572319] Lustre: server umount lustre-MDT0001 complete [ 1231.526224] Lustre: server umount lustre-OST0000 complete [ 1248.770239] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1251.257185] Key type lgssc unregistered [ 1251.557475] LNet: 43803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.564588] LNetError: 43803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.581591] LNet: Removed LNI 192.168.201.136@tcp [ 1252.565478] Key type .llcrypt unregistered [ 1252.566941] Key type ._llcrypt unregistered [ 1273.848502] Key type ._llcrypt registered [ 1273.851786] Key type .llcrypt registered [ 1274.000680] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1275.848249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1275.928160] alg: No test for adler32 (adler32-zlib) [ 1277.164515] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1277.562875] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1279.271235] Key type lgssc registered [ 1280.761663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1291.310900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1291.326831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1292.990358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.685836] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1303.522254] LustreError: 45218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.534055] LustreError: 45218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1307.278562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.712036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1311.885172] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1319.163411] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1323.713628] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1330.900543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1331.274221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.092252] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1338.472925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1345.240992] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1353.004492] Lustre: lustre-OST0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 1353.701229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1353.718692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.738522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1354.211746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1354.230729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.099158] LustreError: 47038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.226803] Lustre: server umount lustre-OST0000 complete [ 1378.857647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.872104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1378.882872] Lustre: Skipped 3 previous similar messages [ 1382.109774] Lustre: server umount lustre-MDT0000 complete [ 1383.905152] LustreError: 45218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.939030] LustreError: 45218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1385.651800] LustreError: 45198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438111 with bad export cookie 11797149681131275002 [ 1385.664908] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1391.966329] Lustre: server umount lustre-MDT0001 complete [ 1400.956353] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1403.535486] Key type lgssc unregistered [ 1403.798910] LNet: 48923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.811738] LNetError: 48923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1403.831662] LNet: Removed LNI 192.168.201.136@tcp [ 1404.526156] Key type .llcrypt unregistered [ 1404.528168] Key type ._llcrypt unregistered [ 1425.260935] Key type ._llcrypt registered [ 1425.263690] Key type .llcrypt registered [ 1425.416136] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1426.550802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1426.758268] alg: No test for adler32 (adler32-zlib) [ 1427.857905] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1428.208364] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1429.895188] Key type lgssc registered [ 1430.991680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1438.243391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1438.263349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1439.700371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1444.039486] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1448.930934] LustreError: 50332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.950429] LustreError: 50332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1452.378942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.722179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1457.363575] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1463.160612] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1467.686657] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1475.190315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1475.673116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.741086] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1484.841104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1489.218493] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1494.191240] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1497.229737] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1509.509577] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 8h23m53s [ 1511.153675] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1514.467881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1514.474594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.500632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1515.556150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1515.577021] Lustre: Skipped 4 previous similar messages [ 1516.843521] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [ 1519.412796] Lustre: server umount lustre-MDT0000 complete [ 1521.637772] LustreError: 50332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1521.665443] LustreError: 50332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1522.461726] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1522.961511] LustreError: 50328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1527.783542] LustreError: 51060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1527.806419] LustreError: 51060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1528.055198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1528.275258] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1528.523864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.609076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1529.560239] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1533.106839] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1536.629820] Lustre: DEBUG MARKER: Wait for df (36672) ... [ 1538.251537] Lustre: DEBUG MARKER: done [ 1544.162178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1544.162348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.162353] Lustre: Skipped 2 previous similar messages [ 1544.163733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1544.163738] Lustre: Skipped 1 previous similar message [ 1544.200124] LustreError: Skipped 1 previous similar message [ 1548.591950] Lustre: server umount lustre-OST0000 complete [ 1555.426574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.440263] Lustre: Skipped 1 previous similar message [ 1555.453851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1555.457104] Lustre: Skipped 4 previous similar messages [ 1566.180435] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1566.487491] Lustre: server umount lustre-MDT0000 complete [ 1568.736423] LustreError: 50331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.744655] LustreError: 50331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1570.257746] LustreError: 51067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438296 with bad export cookie 14218657290266753028 [ 1570.264467] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1570.273994] LustreError: 51067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1570.626479] Lustre: server umount lustre-MDT0001 complete [ 1581.954900] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1585.666982] Key type lgssc unregistered [ 1586.159675] LNet: 55265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.171397] LNetError: 55265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.242052] LNet: Removed LNI 192.168.201.136@tcp [ 1588.849227] Key type .llcrypt unregistered [ 1588.858036] Key type ._llcrypt unregistered [ 1630.291358] Key type ._llcrypt registered [ 1630.293869] Key type .llcrypt registered [ 1630.407993] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1631.860904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1631.876110] alg: No test for adler32 (adler32-zlib) [ 1632.927808] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1633.207664] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1634.991239] Key type lgssc registered [ 1636.280655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1645.423214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1645.436945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.804391] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1646.849355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1647.051709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1651.740566] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1659.936418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1660.152270] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1660.181086] Lustre: 57470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1660.580804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1665.681968] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1672.700637] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1679.690967] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1687.208478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1687.480993] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1687.640376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.468142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1695.553837] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1703.580916] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1711.367527] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 8h27m15s [ 1713.561938] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1721.226987] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1724.218793] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1724.301722] LustreError: 59999:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1744.710150] Lustre: lustre-MDT0000: Client 6acf0376-a4e8-4f08-9971-fc3d13c638f4 (at 192.168.201.36@tcp) reconnecting [ 1764.327195] LustreError: 59999:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1767.450152] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1770.262909] Lustre: DEBUG MARKER: done [ 1776.608407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1776.612833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1776.620072] LustreError: Skipped 1 previous similar message [ 1776.621128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1776.645547] Lustre: Skipped 1 previous similar message [ 1781.748908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1781.762937] Lustre: Skipped 2 previous similar messages [ 1782.217544] Lustre: server umount lustre-OST0000 complete [ 1786.848829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.853555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.864698] Lustre: Skipped 1 previous similar message [ 1786.876840] Lustre: Skipped 1 previous similar message [ 1792.540568] Lustre: server umount lustre-MDT0000 complete [ 1796.562468] LustreError: 57464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438522 with bad export cookie 14547204984772703463 [ 1796.570420] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1796.950208] Lustre: server umount lustre-MDT0001 complete [ 1809.309611] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 1812.696830] Key type lgssc unregistered [ 1812.984568] LNet: 61804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.997960] LNetError: 61804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.065839] LNet: Removed LNI 192.168.201.136@tcp [ 1814.995876] Key type .llcrypt unregistered [ 1814.997808] Key type ._llcrypt unregistered [ 1861.877883] Key type ._llcrypt registered [ 1861.880714] Key type .llcrypt registered [ 1861.987484] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 1862.860402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1862.911500] alg: No test for adler32 (adler32-zlib) [ 1864.040211] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1864.267067] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 1865.967333] Key type lgssc registered [ 1868.047909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1873.752754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1873.767887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1875.025435] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1875.052230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1875.388698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.580067] Lustre: Failing over lustre-MDT0000 [ 1876.838194] Lustre: server umount lustre-MDT0000 complete [ 1883.438880] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:30:08 (1788438608) [ 1890.944283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.568609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.277894] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1906.105911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.182951] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1906.208059] Lustre: 64603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1906.478775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1911.162818] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1918.696884] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1925.108665] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1931.821656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1932.104864] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1932.240232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.548531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1938.603476] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 1945.530547] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1951.160933] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1957.856836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1957.865286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.871448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.393827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1959.395323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.513346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.527500] Lustre: Skipped 4 previous similar messages [ 1969.632749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1969.641218] Lustre: Skipped 2 previous similar messages [ 1972.191395] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1972.557360] Lustre: server umount lustre-MDT0000 complete [ 1977.148019] LustreError: 66458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438703 with bad export cookie 16293254533191844023 [ 1977.149492] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1977.167953] LustreError: 66458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1977.853712] Lustre: server umount lustre-MDT0001 complete [ 1982.300592] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1986.848525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1995.755044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.257151] Lustre: 63526:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788438706/real 1788438706] req@00000000c25ce0c4 x1875313782989952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788438722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1996.289696] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.302327] Lustre: Skipped 1 previous similar message [ 1999.328124] Lustre: 63524:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788438709/real 1788438709] req@00000000e8036fe3 x1875313782990080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788438725 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2001.375619] Lustre: 63524:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788438711/real 1788438711] req@000000003f7dfe1c x1875313782990208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788438727 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2002.465035] LustreError: 63522:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000664145bd x1875313782990848/t0(0) o250->MGC192.168.201.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2002.830468] LustreError: 66451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.854502] LustreError: 66451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2002.948064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.094805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2003.938289] Lustre: 63525:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788438713/real 1788438713] req@00000000cd0eafe1 x1875313782990464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788438729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2004.971054] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2004.973653] LustreError: 66452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2006.061746] LustreError: 68374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2006.097807] LustreError: 68374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2008.438354] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2009.056532] Lustre: 63526:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788438718/real 1788438718] req@00000000c64b1d6f x1875313782990592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788438734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2011.110724] LustreError: 66451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2011.140889] LustreError: 66451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2016.227543] LustreError: 66452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.257541] LustreError: 66452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2018.921343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.492043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2024.946525] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2024.962477] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2033.355398] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2039.663270] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2047.258867] Lustre: server umount lustre-MDT0000 complete [ 2050.016516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2050.026762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2050.049174] LustreError: 66451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.116204] LustreError: 64594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438777 with bad export cookie 16293254533191846298 [ 2051.121315] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2051.130039] LustreError: 64594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2051.528291] Lustre: server umount lustre-MDT0001 complete [ 2058.323817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2068.166112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2071.007706] Lustre: 63523:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788438781/real 1788438781] req@000000001e81b5b8 x1875313783018368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788438797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2071.035792] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.050576] Lustre: Skipped 2 previous similar messages [ 2076.640882] LustreError: 66452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.666732] LustreError: 66452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2076.718482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.753320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2082.181306] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2090.034652] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2091.890184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.658979] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2097.725980] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2105.959939] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2112.516876] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2123.239534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2123.251989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.260471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.280370] Lustre: Skipped 2 previous similar messages [ 2123.304393] Lustre: Skipped 2 previous similar messages [ 2126.877289] Lustre: server umount lustre-MDT0000 complete [ 2128.353177] LustreError: 71055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.364189] LustreError: 71055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2131.044882] LustreError: 66458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438857 with bad export cookie 16293254533191847677 [ 2131.058214] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2131.067370] LustreError: 66458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2131.418788] Lustre: server umount lustre-MDT0001 complete [ 2138.436339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2141.944938] Lustre: DEBUG MARKER: files compared the same [ 2154.448826] Lustre: server umount lustre-OST0000 complete [ 2172.753239] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2175.337220] Key type lgssc unregistered [ 2175.706880] LNet: 74332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2175.713455] LNetError: 74332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2175.725169] LNet: Removed LNI 192.168.201.136@tcp [ 2177.046993] Key type .llcrypt unregistered [ 2177.049526] Key type ._llcrypt unregistered [ 2203.997392] Key type ._llcrypt registered [ 2203.999986] Key type .llcrypt registered [ 2204.085508] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2205.298400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2205.368120] alg: No test for adler32 (adler32-zlib) [ 2206.637128] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2206.873416] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2208.535300] Key type lgssc registered [ 2210.308017] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2216.895375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2216.930031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.503805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2224.379616] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug +malloc all [ 2228.704592] LustreError: 75745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.743652] LustreError: 75745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2233.830819] LustreError: 75746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.079720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.517604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2241.301820] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug +malloc all [ 2249.857513] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2256.010377] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2262.112973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2262.428322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2263.482848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2268.230610] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug +malloc all [ 2276.368598] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2292.205171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.209356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2292.226365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2292.230733] Lustre: Skipped 1 previous similar message [ 2295.631462] Lustre: server umount lustre-OST0000 complete [ 2302.441054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.460171] Lustre: Skipped 2 previous similar messages [ 2302.461069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2302.483260] Lustre: Skipped 1 previous similar message [ 2305.744504] Lustre: server umount lustre-MDT0000 complete [ 2307.551892] LustreError: 75742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.571118] LustreError: 75742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2308.927394] LustreError: 75727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439035 with bad export cookie 4104764984393268200 [ 2308.934043] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2308.934618] LustreError: 75727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2309.133649] Lustre: server umount lustre-MDT0001 complete [ 2317.188462] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2320.241517] Key type lgssc unregistered [ 2320.514770] LNet: 79296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.520890] LNetError: 79296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.532381] LNet: Removed LNI 192.168.201.136@tcp [ 2321.307968] Key type .llcrypt unregistered [ 2321.311775] Key type ._llcrypt unregistered [ 2346.302277] Key type ._llcrypt registered [ 2346.308051] Key type .llcrypt registered [ 2346.470160] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2347.603447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2347.660842] alg: No test for adler32 (adler32-zlib) [ 2348.920672] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2349.156995] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2350.833301] Key type lgssc registered [ 2352.009974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2359.452130] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2359.473855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2375.138172] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2380.255343] LustreError: 80682:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.136@tcp: failed processing log, type 4: rc = -110 [ 2406.943372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.620540] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2421.205806] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2427.612765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2427.725586] Lustre: *** cfs_fail_loc=706, val=0*** [ 2428.056294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2428.109371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2433.241151] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2433.513671] LustreError: 80707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.534117] LustreError: 80707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2438.627908] LustreError: 80708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.544549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2441.698909] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2441.708213] Lustre: Skipped 1 previous similar message [ 2441.822423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2446.573680] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2453.806364] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2459.805156] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2462.176578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2462.186663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2462.211805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2462.217551] Lustre: Skipped 1 previous similar message [ 2467.304853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2468.304853] Lustre: server umount lustre-OST0000 complete [ 2477.542462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.544668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2477.548023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2477.548032] Lustre: Skipped 1 previous similar message [ 2477.554151] Lustre: Skipped 2 previous similar messages [ 2482.664577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2482.669042] Lustre: Skipped 2 previous similar messages [ 2487.263114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2487.439620] Lustre: server umount lustre-MDT0000 complete [ 2487.777770] LustreError: 83177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.803713] LustreError: 83177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2490.772497] LustreError: 80688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439216 with bad export cookie 11998134075531979335 [ 2490.775209] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2490.781158] LustreError: 80688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2491.026586] Lustre: server umount lustre-MDT0001 complete [ 2500.846421] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2503.193111] Key type lgssc unregistered [ 2503.475546] LNet: 84297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2503.480892] LNetError: 84297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2503.497650] LNet: Removed LNI 192.168.201.136@tcp [ 2504.241186] Key type .llcrypt unregistered [ 2504.243168] Key type ._llcrypt unregistered [ 2528.913914] Key type ._llcrypt registered [ 2528.918482] Key type .llcrypt registered [ 2529.076253] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2530.326222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2530.466519] alg: No test for adler32 (adler32-zlib) [ 2531.534915] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2531.823325] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2533.527184] Key type lgssc registered [ 2534.933921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2541.504597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2541.519763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.818579] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2553.028184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.579512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2558.945463] LustreError: 86400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.950825] LustreError: 86400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2559.541385] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2563.040903] LustreError: 86404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.070679] LustreError: 86404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2568.166176] LustreError: 86403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.212475] LustreError: 86403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2570.812722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2571.374490] LustreError: 86398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2571.421753] LustreError: 86398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2571.462313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.889059] LustreError: 86404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.937225] LustreError: 86404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2579.743311] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2588.132577] LustreError: 87147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.153413] LustreError: 87147:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2590.675591] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2590.720269] Lustre: MGS: Client 51bc07b9-4eb9-42dc-8143-f4d427e5f7cf (at 0@lo) reconnecting [ 2590.773118] LustreError: 86414:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2591.219716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2591.320244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2595.819955] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2611.686881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2611.701188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.704709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2611.718109] Lustre: Skipped 1 previous similar message [ 2611.743113] Lustre: Skipped 1 previous similar message [ 2615.633630] Lustre: server umount lustre-OST0000 complete [ 2621.923158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.924938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2621.929286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2621.935527] LustreError: Skipped 1 previous similar message [ 2621.952663] Lustre: Skipped 1 previous similar message [ 2625.803230] Lustre: server umount lustre-MDT0000 complete [ 2627.044036] LustreError: 86400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.059099] LustreError: 86400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2629.838055] Lustre: server umount lustre-MDT0001 complete [ 2633.863849] Lustre: server umount MGS complete [ 2646.317837] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 08:42:50 (1788439370) [ 2674.919454] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 2684.855717] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2718.431956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.294559] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2731.713742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.012086] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2732.049623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2732.120950] Lustre: lustre-MDT0001: new disk, initializing [ 2732.242555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2732.304928] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2732.308296] Lustre: 93569:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2732.328542] LustreError: 93569:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2736.693102] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2747.264711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2747.479252] Lustre: lustre-OST0000: new disk, initializing [ 2747.483248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2747.489589] Lustre: 94335:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2747.570137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2754.737280] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2758.764202] Lustre: 94793:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000001eef9464 x1875314482390656/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788439500 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2764.507937] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2771.416541] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.430307] Lustre: MGS: Client 0324e3a1-a80d-49c8-b770-df103c9f20be (at 0@lo) reconnecting [ 2771.475698] Lustre: 95382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2771.488250] LustreError: 93555:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2771.518955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2771.584758] Lustre: lustre-MDT0000: new disk, initializing [ 2771.714416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2771.748472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2771.907018] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2772.840441] Lustre: 95406:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f36b9a93 x1875314482397184/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788439514 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2776.930503] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2779.103781] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2780.983634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2780.995838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2782.119826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2782.132982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2782.170534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2801.632191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2801.636815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2801.653566] Lustre: Skipped 1 previous similar message [ 2801.668630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2806.250239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2806.262950] Lustre: Skipped 2 previous similar messages [ 2807.583745] Lustre: server umount lustre-OST0000 complete [ 2816.485736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2816.492816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2816.521400] LustreError: Skipped 1 previous similar message [ 2816.531828] Lustre: Skipped 1 previous similar message [ 2816.539871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2816.543668] Lustre: Skipped 1 previous similar message [ 2817.899098] Lustre: server umount lustre-MDT0000 complete [ 2821.600656] LustreError: 93539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.621950] LustreError: 93539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2828.238384] Lustre: server umount lustre-MDT0001 complete [ 2832.023476] Lustre: server umount MGS complete [ 2843.842230] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 08:46:08 (1788439568) [ 2861.935900] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 2864.931689] Key type lgssc unregistered [ 2865.222776] LNet: 97908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2865.226945] LNetError: 97908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2865.251829] LNet: Removed LNI 192.168.201.136@tcp [ 2866.141283] Key type .llcrypt unregistered [ 2866.143678] Key type ._llcrypt unregistered [ 2885.743822] Key type ._llcrypt registered [ 2885.745909] Key type .llcrypt registered [ 2885.907465] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 2887.575735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2887.806967] alg: No test for adler32 (adler32-zlib) [ 2889.044643] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2889.483884] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 2891.175174] Key type lgssc registered [ 2892.924478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2902.243833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2902.293789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2903.778064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.787540] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2909.162142] LustreError: 99126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.174381] LustreError: 99126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2913.257929] LustreError: 99127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2917.598188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.922529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2921.999249] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2930.534339] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2936.663799] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2943.154224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2943.471110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2948.619119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2950.718193] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 2957.828232] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2969.057538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2969.070042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.084098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2972.133801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2972.139365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2972.148121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.196884] Lustre: Skipped 1 previous similar message [ 2975.154766] Lustre: server umount lustre-OST0000 complete [ 2979.302334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2979.308626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2979.317022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2982.368706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2982.369323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2985.821477] Lustre: server umount lustre-MDT0000 complete [ 2987.490538] LustreError: 99127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.505066] LustreError: 99127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2989.895358] LustreError: 99107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439716 with bad export cookie 4457994322495313651 [ 2989.911953] LustreError: 99107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2989.912215] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2990.280796] Lustre: server umount lustre-MDT0001 complete [ 3003.360965] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3006.294989] Key type lgssc unregistered [ 3006.681761] LNet: 102658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3006.694595] LNetError: 102658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3006.718907] LNet: Removed LNI 192.168.201.136@tcp [ 3007.634901] Key type .llcrypt unregistered [ 3007.636624] Key type ._llcrypt unregistered [ 3025.868908] Key type ._llcrypt registered [ 3025.871571] Key type .llcrypt registered [ 3026.005510] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3027.046114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3027.186219] alg: No test for adler32 (adler32-zlib) [ 3028.601442] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3028.967748] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3030.711313] Key type lgssc registered [ 3032.144906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3038.405458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3038.413382] LustreError: 103736:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3038.424291] LustreError: 103739:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3038.428919] LustreError: 103736:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 3038.437251] LustreError: 103736:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3038.444020] LustreError: 103736:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3038.462257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.117595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.503538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.754568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3053.346144] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3059.400254] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing lsmod [ 3062.662653] LustreError: 104939:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3062.679604] LustreError: 104946:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3062.689755] LustreError: 104939:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 3062.699301] LustreError: 104939:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3062.708338] LustreError: 104939:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3062.708600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3063.132344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.215784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3069.422751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3069.434891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.446953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3070.435575] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.436607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3070.460354] Lustre: Skipped 1 previous similar message [ 3070.480016] Lustre: Skipped 2 previous similar messages [ 3073.569809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3073.584547] Lustre: Skipped 2 previous similar messages [ 3078.627704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3078.634479] Lustre: Skipped 2 previous similar messages [ 3080.159223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3080.289132] Lustre: server umount lustre-MDT0000 complete [ 3083.414593] LustreError: 103754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439809 with bad export cookie 15903127976261414052 [ 3083.419578] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3083.429157] LustreError: 103754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3083.749063] LustreError: 103772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.752020] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3083.761655] LustreError: 103772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3083.778418] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3083.831699] LustreError: 104208:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000186a8ef2 x1875315004561664/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3084.053390] Lustre: server umount lustre-MDT0001 complete [ 3089.954139] Lustre: server umount lustre-OST0000 complete [ 3100.667551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.290862] LustreError: 106010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3101.424739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3106.294843] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3106.785775] LustreError: 106011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.891872] LustreError: 106010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.000814] LustreError: 106011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.079771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.815940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3124.764586] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3134.374598] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3140.834449] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3147.306568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3147.623716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3153.811959] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3164.491571] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3171.235977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3183.587453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3183.598280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3183.622831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3186.670410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3188.793817] Lustre: server umount lustre-OST0000 complete [ 3195.872680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3195.888882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.925189] Lustre: Skipped 1 previous similar message [ 3200.239710] Lustre: server umount lustre-MDT0000 complete [ 3202.027259] LustreError: 106010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3202.041233] LustreError: 106010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3204.668455] LustreError: 105991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439930 with bad export cookie 15903127976261414976 [ 3204.681143] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3204.692565] LustreError: 105991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3205.040394] Lustre: server umount lustre-MDT0001 complete [ 3216.874217] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3219.728155] Key type lgssc unregistered [ 3220.125460] LNet: 109568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3220.132774] LNetError: 109568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3220.145504] LNet: Removed LNI 192.168.201.136@tcp [ 3220.934107] Key type .llcrypt unregistered [ 3220.937377] Key type ._llcrypt unregistered [ 3244.771787] Key type ._llcrypt registered [ 3244.773287] Key type .llcrypt registered [ 3244.885662] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3246.156374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3246.177108] alg: No test for adler32 (adler32-zlib) [ 3247.302494] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3247.633121] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3249.351413] Key type lgssc registered [ 3250.572949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3259.950716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3260.001657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.419922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.030917] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3271.649274] LustreError: 110982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.699299] LustreError: 110982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3276.008127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3276.516833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3282.545585] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3291.811336] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3297.552535] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3305.833326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3306.417517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.501112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3315.679315] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3325.818367] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3331.062310] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3335.388323] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3342.308172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3342.315935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.324963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3342.356681] Lustre: Skipped 1 previous similar message [ 3345.988867] Lustre: server umount lustre-OST0000 complete [ 3352.559402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.598624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.601168] Lustre: Skipped 1 previous similar message [ 3356.222948] Lustre: server umount lustre-MDT0000 complete [ 3357.674196] LustreError: 110982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.691971] LustreError: 110982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3361.450074] LustreError: 110964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440087 with bad export cookie 8869304114980444816 [ 3361.465104] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3361.475135] LustreError: 110964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3362.489279] Lustre: server umount lustre-MDT0001 complete [ 3375.148701] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3378.115733] Key type lgssc unregistered [ 3378.388686] LNet: 114617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3378.395112] LNetError: 114617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3379.446752] LNet: Removed LNI 192.168.201.136@tcp [ 3380.639103] Key type .llcrypt unregistered [ 3380.645865] Key type ._llcrypt unregistered [ 3401.480962] Key type ._llcrypt registered [ 3401.483856] Key type .llcrypt registered [ 3401.593970] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3402.937925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3402.963311] alg: No test for adler32 (adler32-zlib) [ 3404.199911] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3404.531812] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3406.266425] Key type lgssc registered [ 3407.713619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3416.776560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3416.817424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3418.299902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3424.201567] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3428.837055] LustreError: 115837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3428.845893] LustreError: 115837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3433.952993] LustreError: 115838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3433.992610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3434.548956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3440.097794] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3449.344783] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3455.679767] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3463.765399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3464.377291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.871592] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3480.476792] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3487.925859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3500.514967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3500.518945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3500.528980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.586603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3503.592909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.605822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.619208] Lustre: Skipped 1 previous similar message [ 3505.463767] Lustre: server umount lustre-OST0000 complete [ 3511.282579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3511.294528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3511.318723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.844576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.846810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.864935] Lustre: Skipped 1 previous similar message [ 3516.244744] Lustre: server umount lustre-MDT0000 complete [ 3518.963328] LustreError: 116782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.993690] LustreError: 116782:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3520.521672] LustreError: 115820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440246 with bad export cookie 9457484469792542266 [ 3520.528460] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3520.535584] LustreError: 115820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3520.915626] Lustre: server umount lustre-MDT0001 complete [ 3536.567562] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3540.938498] Key type lgssc unregistered [ 3541.291921] LNet: 119393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3541.321158] LNetError: 119393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3541.341299] LNet: Removed LNI 192.168.201.136@tcp [ 3542.632080] Key type .llcrypt unregistered [ 3542.637071] Key type ._llcrypt unregistered [ 3568.007237] Key type ._llcrypt registered [ 3568.009670] Key type .llcrypt registered [ 3568.157666] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3569.629418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3569.699486] alg: No test for adler32 (adler32-zlib) [ 3570.848872] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3571.172076] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3572.847832] Key type lgssc registered [ 3574.013270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3583.273465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3583.289593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3584.860225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3590.116817] LustreError: 120807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3590.137377] LustreError: 120807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3590.676188] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3594.212109] LustreError: 120808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3599.330484] LustreError: 120807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3600.217158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3600.810514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3606.445762] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3614.687466] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3621.000515] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3628.340536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3628.816270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.876465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3636.154279] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3644.687690] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3651.010703] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3654.770841] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3654.778519] Lustre: Skipped 2 previous similar messages [ 3658.332882] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3658.354153] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3659.490701] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3659.496990] Lustre: Skipped 2 previous similar messages [ 3667.624898] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3667.635718] Lustre: Skipped 1 previous similar message [ 3673.774171] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.36@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3673.785656] Lustre: Skipped 2 previous similar messages [ 3673.956263] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.36@tcp 192.168.0.[2,10]@tcp [ 3684.116244] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.36/24@tcp in log lustre-MDT0000 [ 3684.126749] Lustre: Skipped 2 previous similar messages [ 3694.151635] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.36/24@tcp [ 3694.158259] Lustre: Skipped 1 previous similar message [ 3701.729621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.733292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3701.750115] LustreError: Skipped 1 previous similar message [ 3701.751657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3701.778707] Lustre: Skipped 1 previous similar message [ 3703.183203] Lustre: server umount lustre-OST0000 complete [ 3708.171593] Lustre: server umount lustre-MDT0000 complete [ 3708.384969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3708.390737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3708.408478] Lustre: Skipped 1 previous similar message [ 3708.419297] LustreError: 120807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3711.977831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.132865] LustreError: 120789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440439 with bad export cookie 9669507606605733085 [ 3713.135250] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3713.141109] LustreError: 120789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3715.605331] Lustre: server umount lustre-MDT0001 complete [ 3728.369803] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3730.858500] Key type lgssc unregistered [ 3731.251213] LNet: 125804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3731.260989] LNetError: 125804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3731.278825] LNet: Removed LNI 192.168.201.136@tcp [ 3732.132798] Key type .llcrypt unregistered [ 3732.138701] Key type ._llcrypt unregistered [ 3762.404750] Key type ._llcrypt registered [ 3762.407748] Key type .llcrypt registered [ 3762.604487] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3764.046147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3764.073586] alg: No test for adler32 (adler32-zlib) [ 3765.290887] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3765.663543] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3767.401546] Key type lgssc registered [ 3768.759251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3777.910683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3777.922365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3779.412381] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3779.466473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3779.469860] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.36/24@tcp [ 3784.468370] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3789.796201] LustreError: 127386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3789.827485] LustreError: 127386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3792.964754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3793.209675] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3793.231567] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.36/24@tcp [ 3793.328285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3798.521601] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3807.131731] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3813.564488] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3820.250595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3820.551121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3824.706788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3827.338057] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3835.346505] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3850.209319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3850.225661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.247384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3853.286158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3853.286269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3853.317355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3854.593479] Lustre: server umount lustre-OST0000 complete [ 3859.939328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3859.952766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3859.966724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3859.974764] Lustre: Skipped 1 previous similar message [ 3863.522191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3863.530361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3864.972218] Lustre: server umount lustre-MDT0000 complete [ 3868.643796] LustreError: 127386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3868.656170] LustreError: 127386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3868.874895] LustreError: 127368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440595 with bad export cookie 14868396114493237455 [ 3868.885463] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3868.889423] LustreError: 127368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3869.410980] Lustre: server umount lustre-MDT0001 complete [ 3880.337691] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 3883.181825] Key type lgssc unregistered [ 3883.481246] LNet: 130973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3883.489961] LNetError: 130973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3884.518778] LNet: Removed LNI 192.168.201.136@tcp [ 3885.410890] Key type .llcrypt unregistered [ 3885.413972] Key type ._llcrypt unregistered [ 3911.146814] Key type ._llcrypt registered [ 3911.149223] Key type .llcrypt registered [ 3911.301571] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 3926.307062] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 3927.428460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3927.599771] alg: No test for adler32 (adler32-zlib) [ 3928.698759] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3928.956314] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 3930.679968] Key type lgssc registered [ 3932.130788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3967.754751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3967.785678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3969.088805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3969.156505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3969.275450] Lustre: lustre-MDT0000: new disk, initializing [ 3969.403230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.416541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3974.242256] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 3984.478165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3984.590994] Lustre: 134368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3984.670889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3984.674180] Lustre: Skipped 1 previous similar message [ 3984.736468] Lustre: lustre-MDT0001: new disk, initializing [ 3984.830867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3984.857982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3984.873260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3990.093769] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4000.356353] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4005.992435] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4013.095850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4013.409204] Lustre: lustre-OST0000: new disk, initializing [ 4013.413916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4013.424562] Lustre: 136335:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4013.520895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4020.496202] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4020.867309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4020.883767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4021.006638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4029.916618] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4035.117641] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4035.560739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4041.063419] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4041.303320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4046.817390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4046.818072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.833014] LustreError: Skipped 1 previous similar message [ 4046.854269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4051.947189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4051.958569] Lustre: Skipped 2 previous similar messages [ 4057.070614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4058.079184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4058.212991] Lustre: server umount lustre-OST0000 complete [ 4062.178441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4062.183335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4062.190822] Lustre: Skipped 2 previous similar messages [ 4062.207350] Lustre: Skipped 2 previous similar messages [ 4067.300717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4067.306603] Lustre: Skipped 1 previous similar message [ 4068.531902] Lustre: server umount lustre-MDT0000 complete [ 4072.177233] LustreError: 134360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440798 with bad export cookie 12407341256137397149 [ 4072.183260] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4072.190037] LustreError: 134360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4072.707954] Lustre: server umount lustre-MDT0001 complete [ 4082.798556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4083.242309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4088.293591] LustreError: 138557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.316593] LustreError: 138557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4089.832743] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4093.410406] LustreError: 138558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.537234] LustreError: 138557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4099.916800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.345851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4105.624623] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4113.738158] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4119.170304] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4126.071461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4126.647257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4132.424484] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4139.440424] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4144.581871] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4144.899945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4149.766244] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4150.083303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4159.083545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4159.213468] Lustre: lustre-OST0001: new disk, initializing [ 4159.215864] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4159.220818] Lustre: 141927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4159.293241] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4166.003456] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4166.360112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4166.377275] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4166.541986] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4175.297756] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4175.519481] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4180.609730] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4180.881488] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4186.821335] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90c7061c2000.ost_server_uuid 50 [ 4188.765537] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90c7061c2000.ost_server_uuid in IDLE state after 0 sec [ 4207.585282] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4207.587262] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.611501] LustreError: Skipped 1 previous similar message [ 4207.613148] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4207.644628] Lustre: Skipped 1 previous similar message [ 4217.311283] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4217.517357] Lustre: server umount lustre-OST0001 complete [ 4217.843097] LustreError: 142505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4217.872943] LustreError: 142505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4222.944676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.961940] LustreError: 140379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4222.971334] LustreError: 140379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4228.075454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4228.079718] Lustre: Skipped 6 previous similar messages [ 4233.187963] LustreError: 142307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.208952] LustreError: 142307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4236.257152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4236.649882] Lustre: server umount lustre-OST0000 complete [ 4243.429915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4243.450270] Lustre: Skipped 2 previous similar messages [ 4247.512610] Lustre: server umount lustre-MDT0000 complete [ 4252.192621] LustreError: 138539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440978 with bad export cookie 12407341256137398031 [ 4252.196453] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4252.222634] LustreError: 138539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4252.643377] Lustre: server umount lustre-MDT0001 complete [ 4273.267206] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 4276.103392] Key type lgssc unregistered [ 4276.323843] LNet: 144956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4276.331600] LNetError: 144956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4277.349807] LNet: Removed LNI 192.168.201.136@tcp [ 4278.118887] Key type .llcrypt unregistered [ 4278.120703] Key type ._llcrypt unregistered [ 4320.874967] Key type ._llcrypt registered [ 4320.877344] Key type .llcrypt registered [ 4320.970331] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 4340.610830] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4342.539053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4342.715369] alg: No test for adler32 (adler32-zlib) [ 4344.016617] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4344.549083] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 4346.311303] Key type lgssc registered [ 4348.329623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4384.078114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4384.100510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4385.428668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4385.503626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4385.577825] Lustre: lustre-MDT0000: new disk, initializing [ 4385.695195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4385.716825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4390.280631] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4401.658219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4401.793276] Lustre: 149115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4401.881784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4401.886267] Lustre: Skipped 1 previous similar message [ 4401.950559] Lustre: lustre-MDT0001: new disk, initializing [ 4402.034967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4402.079383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4402.087401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4406.320973] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4415.506888] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4420.872748] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4426.841804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4427.078691] Lustre: lustre-OST0000: new disk, initializing [ 4427.082834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4427.090062] Lustre: 151084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4427.195842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4434.376280] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4437.625068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4437.649666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4437.807376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4445.193816] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4450.512978] Lustre: Failing over lustre-OST0000 [ 4450.729733] Lustre: server umount lustre-OST0000 complete [ 4467.685777] Lustre: 147336:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441177/real 1788441177] req@000000005f32018f x1875316383449600/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788441193 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4467.702440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.665817] Lustre: 147334:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441179/real 1788441179] req@00000000a2296f7a x1875316383450880/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788441195 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4469.729562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4470.818953] Lustre: 147335:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441179/real 1788441179] req@000000001801d1ba x1875316383449728/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788441195 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4470.851742] Lustre: 147335:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4471.808984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4472.094970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.118693] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4475.106254] Lustre: 147333:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441184/real 1788441184] req@00000000ddc4b7d6 x1875316383451904/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788441200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4475.142448] Lustre: 147333:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4477.233609] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4479.268132] Lustre: 147336:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441189/real 1788441189] req@000000002f1c17fe x1875316383452800/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788441205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4479.384782] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4479.386159] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4479.716883] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4484.388986] Lustre: Failing over lustre-MDT0000 [ 4484.661278] Lustre: server umount lustre-MDT0000 complete [ 4487.652982] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4487.680794] Lustre: Skipped 1 previous similar message [ 4492.772323] LustreError: 149126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4492.802618] LustreError: 149126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4494.098785] LustreError: 149120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4497.889359] LustreError: 149122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4497.903931] LustreError: 149122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4502.955719] Lustre: 147336:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441213/real 1788441213] req@00000000580bf87c x1875316383465728/t0(0) o400->MGC192.168.201.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788441229 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4502.996661] Lustre: 147336:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4503.009426] LustreError: 149126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4503.013161] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4503.017793] LustreError: 149126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4504.597284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.130310] LustreError: 149121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.172493] LustreError: 149121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4513.250069] LustreError: 147332:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b88ec814 x1875316383471232/t0(0) o250->MGC192.168.201.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4513.631290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4513.712571] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4515.598936] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4518.892308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4518.900136] Lustre: Skipped 1 previous similar message [ 4518.934890] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4518.981658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4519.284788] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4529.632490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4529.644628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4529.685616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4530.656091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4530.660076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4532.193965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4532.205605] Lustre: Skipped 1 previous similar message [ 4534.484874] Lustre: server umount lustre-OST0000 complete [ 4539.360418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4539.376654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.390531] Lustre: Skipped 1 previous similar message [ 4539.397109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4544.433570] Lustre: server umount lustre-MDT0000 complete [ 4547.553841] LustreError: 149125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4547.563724] LustreError: 149125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4548.804951] LustreError: 151083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441274 with bad export cookie 14893956238461932212 [ 4548.811619] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4548.813155] LustreError: 151083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4549.018623] Lustre: server umount lustre-MDT0001 complete [ 4560.610759] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 4563.638355] Key type lgssc unregistered [ 4564.048366] LNet: 154740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4564.067433] LNetError: 154740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4565.095534] LNet: Removed LNI 192.168.201.136@tcp [ 4566.144602] Key type .llcrypt unregistered [ 4566.147229] Key type ._llcrypt unregistered [ 4607.408832] Key type ._llcrypt registered [ 4607.410884] Key type .llcrypt registered [ 4607.523555] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4608.857368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4608.970143] alg: No test for adler32 (adler32-zlib) [ 4610.045707] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4610.308240] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 4612.031220] Key type lgssc registered [ 4613.659803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4622.727316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4622.759203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.324116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.334404] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4634.598423] LustreError: 156258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4634.617864] LustreError: 156258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4639.479563] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4639.713867] LustreError: 156259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4643.622372] LustreError: 156287:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4643.634711] LustreError: 156287:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4643.645383] LustreError: 156287:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4643.957895] Lustre: server umount lustre-MDT0000 complete [ 4660.511298] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 4671.808189] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 4708.886123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4709.241726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4709.270555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4709.419327] Lustre: lustre-MDT0000: new disk, initializing [ 4709.549537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4709.581415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4715.394798] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4726.290638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4726.392426] Lustre: 159593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4726.449484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4726.452785] Lustre: Skipped 1 previous similar message [ 4726.552567] Lustre: lustre-MDT0001: new disk, initializing [ 4726.663481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4726.709058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4726.719869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4732.232839] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4745.287274] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4751.231196] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4758.897674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4759.102743] Lustre: lustre-OST0000: new disk, initializing [ 4759.105844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4759.113685] Lustre: 161561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4759.220178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4765.407797] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 4766.804654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4766.818198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4766.934167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4776.388251] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4781.722682] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4783.668429] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:18:28 (1788441508) [ 5028.320909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.322411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5028.352705] Lustre: Skipped 1 previous similar message [ 5028.381146] Lustre: Skipped 2 previous similar messages [ 5031.472288] Lustre: server umount lustre-MDT0000 complete [ 5033.445176] LustreError: 161227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5033.460917] LustreError: 161227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5035.443714] LustreError: 161560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441761 with bad export cookie 15679538400134004423 [ 5035.457403] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5035.459800] LustreError: 161560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5038.560205] LustreError: 159599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5038.563193] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5038.599858] Lustre: Skipped 1 previous similar message [ 5038.615881] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5040.610865] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5042.001592] Lustre: server umount lustre-MDT0001 complete [ 5053.085781] Lustre: server umount lustre-OST0000 complete [ 5064.342330] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 5076.232217] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5119.640655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5119.912556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5119.985966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5120.157957] Lustre: lustre-MDT0000: new disk, initializing [ 5120.283808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5120.325185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5125.560308] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5137.808942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5138.004155] Lustre: 166223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5138.108796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5138.112955] Lustre: Skipped 1 previous similar message [ 5138.215915] Lustre: lustre-MDT0001: new disk, initializing [ 5138.345439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5138.392905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5138.403853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5144.112632] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5154.291392] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5160.826692] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5168.006253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5168.342035] Lustre: lustre-OST0000: new disk, initializing [ 5168.349814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5168.358383] Lustre: 168191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5168.452222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5174.904332] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5175.884946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5175.891734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5176.013101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5184.922807] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5190.069744] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5190.420694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5196.222588] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5196.633476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5201.909575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5201.910866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5201.917412] LustreError: Skipped 1 previous similar message [ 5201.918216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5201.956408] Lustre: Skipped 1 previous similar message [ 5207.026433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5207.056478] Lustre: Skipped 2 previous similar messages [ 5213.151530] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5213.313237] Lustre: server umount lustre-OST0000 complete [ 5217.248706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5217.268337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5217.277602] Lustre: Skipped 3 previous similar messages [ 5223.622301] Lustre: server umount lustre-MDT0000 complete [ 5227.489471] LustreError: 168198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5227.502925] LustreError: 168198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5227.564978] LustreError: 166216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441953 with bad export cookie 15679538400134163246 [ 5227.572622] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5227.575022] LustreError: 166216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5228.209417] Lustre: server umount lustre-MDT0001 complete [ 5238.357418] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:26:02 (1788441962) [ 5257.319274] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 5268.738228] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5305.518985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5305.749804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5305.769160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5305.839957] Lustre: lustre-MDT0000: new disk, initializing [ 5305.898885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5305.912687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5309.521708] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5320.813833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.953376] Lustre: 172951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5321.287048] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5326.474806] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5334.741499] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5342.087575] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5350.349775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5350.639802] Lustre: lustre-OST0000: new disk, initializing [ 5350.642072] Lustre: Skipped 1 previous similar message [ 5350.645767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5350.650153] Lustre: Skipped 2 previous similar messages [ 5350.652895] Lustre: 174927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5358.892776] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5359.251064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5359.255553] Lustre: Skipped 1 previous similar message [ 5359.260118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5359.377943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5370.919735] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5380.070297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5380.074143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5380.075646] LustreError: Skipped 1 previous similar message [ 5380.078267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5380.078274] Lustre: Skipped 2 previous similar messages [ 5380.110281] Lustre: Skipped 2 previous similar messages [ 5385.572723] Lustre: server umount lustre-OST0000 complete [ 5390.312569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5390.331610] Lustre: Skipped 1 previous similar message [ 5395.732364] Lustre: server umount lustre-MDT0000 complete [ 5399.286223] LustreError: 172941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442125 with bad export cookie 15679538400134164135 [ 5399.294103] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5399.723350] Lustre: server umount lustre-MDT0001 complete [ 5409.252065] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:28:53 (1788442133) [ 5428.038643] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 5439.430396] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5479.272779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5479.541840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5479.578226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5479.662278] Lustre: lustre-MDT0000: new disk, initializing [ 5479.770854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5479.774241] Lustre: Skipped 2 previous similar messages [ 5479.813182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5484.649330] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5495.798472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.916550] Lustre: 179253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5496.228623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5500.801632] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5512.192688] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5518.429236] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5525.872143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5526.190675] Lustre: 181230:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5531.798948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5531.985701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5534.027048] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5544.233488] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5552.611444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5552.622137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5552.642205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5552.647022] Lustre: Skipped 8 previous similar messages [ 5557.115813] Lustre: server umount lustre-OST0000 complete [ 5562.848976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5562.852933] LustreError: Skipped 1 previous similar message [ 5567.964124] Lustre: server umount lustre-MDT0000 complete [ 5567.978605] LustreError: 180343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5568.004185] LustreError: 180343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5571.726450] LustreError: 179246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442297 with bad export cookie 15679538400134165220 [ 5571.735673] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5571.743262] LustreError: 179246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5572.138824] Lustre: server umount lustre-MDT0001 complete [ 5582.572137] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 5585.176831] Key type lgssc unregistered [ 5585.461691] LNet: 183085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5585.469724] LNetError: 183085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5585.496815] LNet: Removed LNI 192.168.201.136@tcp [ 5586.421869] Key type .llcrypt unregistered [ 5586.423571] Key type ._llcrypt unregistered [ 5613.438737] Key type ._llcrypt registered [ 5613.441809] Key type .llcrypt registered [ 5613.567889] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5615.668342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5615.766295] alg: No test for adler32 (adler32-zlib) [ 5616.773251] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5617.022777] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 5618.671167] Key type lgssc registered [ 5619.938388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5629.454199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5629.482221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5631.244758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5636.825595] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5641.707630] LustreError: 184498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.739235] LustreError: 184498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5646.825428] LustreError: 184499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5647.519587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5647.998236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5653.465167] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5662.017733] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5667.574157] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5674.029271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5674.289267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5676.335808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5680.763314] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5688.474280] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5712.356614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5712.373596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.393411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5714.400544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5714.414844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5714.422616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5714.447742] Lustre: Skipped 1 previous similar message [ 5716.141617] Lustre: server umount lustre-OST0000 complete [ 5724.641223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5724.647588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5724.678292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5724.680107] Lustre: Skipped 1 previous similar message [ 5724.702812] Lustre: Skipped 1 previous similar message [ 5727.333253] Lustre: server umount lustre-MDT0000 complete [ 5729.771966] LustreError: 184495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5729.797103] LustreError: 184495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5731.441084] LustreError: 184479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442457 with bad export cookie 924454353271776490 [ 5731.445444] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5731.454302] LustreError: 184479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5731.952652] Lustre: server umount lustre-MDT0001 complete [ 5745.162449] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 5748.095962] Key type lgssc unregistered [ 5748.433805] LNet: 188056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5748.439367] LNetError: 188056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5748.453955] LNet: Removed LNI 192.168.201.136@tcp [ 5749.354188] Key type .llcrypt unregistered [ 5749.364058] Key type ._llcrypt unregistered [ 5772.390692] Key type ._llcrypt registered [ 5772.393757] Key type .llcrypt registered [ 5772.539209] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 5773.702664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5773.754316] alg: No test for adler32 (adler32-zlib) [ 5774.875923] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5775.193856] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 5776.855198] Key type lgssc registered [ 5777.946400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5785.609360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5785.634907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.122191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.003885] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5797.362202] LustreError: 189468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5797.378869] LustreError: 189468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5798.372890] LustreError: 189469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.707048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.089363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5806.230059] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5813.372949] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5818.960751] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5826.064701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5826.487364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5833.194959] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5833.654722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5841.125276] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5851.111066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.117745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5851.126681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5853.306021] Lustre: server umount lustre-OST0000 complete [ 5860.234685] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90c7037d2000.ost_server_uuid 50 [ 5874.138174] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5891.818045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.830622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5891.842230] Lustre: Skipped 2 previous similar messages [ 5891.853867] Lustre: Skipped 1 previous similar message [ 5895.170393] Lustre: server umount lustre-MDT0000 complete [ 5897.187208] LustreError: 189468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5897.208859] LustreError: 189468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5899.409860] LustreError: 189448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442625 with bad export cookie 10948373811283610021 [ 5899.431246] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5900.044786] Lustre: server umount lustre-MDT0001 complete [ 5908.091528] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:37:12 (1788442632) [ 5913.565460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5913.944291] LustreError: 193402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5914.004359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5919.203014] LustreError: 193403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5919.258327] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5926.974231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5927.178736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5931.219949] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5938.922354] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5945.105534] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5953.430355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5953.928814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5959.025839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5961.216319] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5969.745459] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5977.508140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5977.874734] Lustre: lustre-OST0001: new disk, initializing [ 5977.882613] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5977.893386] Lustre: 196244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5978.061215] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5985.813486] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 5987.440866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5987.460372] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5987.554072] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5995.946539] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6000.101800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6000.113157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6000.135969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6000.147447] Lustre: Skipped 1 previous similar message [ 6003.178517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6003.185802] Lustre: Skipped 1 previous similar message [ 6003.188597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.203624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6004.753916] Lustre: server umount lustre-OST0000 complete [ 6006.548914] LustreError: 195225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6006.601248] LustreError: 195225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6011.810274] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6012.019432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6017.550685] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6017.757206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6023.654190] LustreError: 196687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6023.670374] LustreError: 196687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6033.889353] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6033.904826] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6033.913186] Lustre: Skipped 1 previous similar message [ 6033.914557] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6033.923430] Lustre: Skipped 2 previous similar messages [ 6037.883406] Lustre: server umount lustre-OST0001 complete [ 6044.128786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6044.137276] Lustre: Skipped 1 previous similar message [ 6044.141434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6044.147147] Lustre: Skipped 1 previous similar message [ 6047.822490] Lustre: server umount lustre-MDT0000 complete [ 6051.907985] LustreError: 196243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442778 with bad export cookie 10948373811283611085 [ 6051.909785] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6051.925968] LustreError: 196243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6052.230581] Lustre: server umount lustre-MDT0001 complete [ 6085.067608] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:40:09 (1788442809) [ 6092.497990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6092.740202] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6092.765858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6093.013944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6099.045660] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6109.192539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6109.242731] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6109.261973] Lustre: 199686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6115.074989] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6124.182880] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6130.662344] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6137.785456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6138.079230] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6144.229039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6145.042383] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6153.138803] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6160.761049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6160.849769] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6161.064819] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6161.076653] Lustre: Skipped 2 previous similar messages [ 6168.230517] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6176.300290] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6179.809748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6179.812995] LustreError: Skipped 1 previous similar message [ 6179.816961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6179.834090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6193.119844] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6193.331664] Lustre: server umount lustre-OST0000 complete [ 6197.221128] LustreError: 201540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6197.248573] LustreError: 201540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6212.065183] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6212.080286] LustreError: Skipped 1 previous similar message [ 6212.087321] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6212.115156] Lustre: Skipped 1 previous similar message [ 6212.127905] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6212.130651] Lustre: Skipped 10 previous similar messages [ 6216.573283] Lustre: server umount lustre-OST0001 complete [ 6227.263647] Lustre: server umount lustre-MDT0000 complete [ 6231.216526] LustreError: 199678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442957 with bad export cookie 10948373811283612499 [ 6231.222359] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6231.235098] LustreError: 199678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6231.519725] Lustre: server umount lustre-MDT0001 complete [ 6258.562628] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:43:03 (1788442983) [ 6277.061130] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 6288.097700] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 6329.212024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.464751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6329.482454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6329.565835] Lustre: lustre-MDT0000: new disk, initializing [ 6329.622602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6329.641836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6334.879172] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6346.257774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.368777] Lustre: 207790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6346.428299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6346.434097] Lustre: Skipped 1 previous similar message [ 6346.491651] Lustre: lustre-MDT0001: new disk, initializing [ 6346.577572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6346.581765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6351.221909] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6360.322686] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6366.061624] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6372.901465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6373.213135] Lustre: lustre-OST0000: new disk, initializing [ 6373.218624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6373.227501] Lustre: 209759:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6379.661249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6379.687966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6379.812177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6380.257597] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6391.752564] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6398.185839] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6398.495860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6404.120721] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6404.359799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6410.721286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6410.732405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6410.735867] LustreError: Skipped 2 previous similar messages [ 6410.736723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6410.736729] Lustre: Skipped 3 previous similar messages [ 6410.782484] Lustre: Skipped 4 previous similar messages [ 6412.920726] Lustre: server umount lustre-OST0000 complete [ 6424.563931] Lustre: server umount lustre-MDT0000 complete [ 6426.080857] LustreError: 209766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6426.106964] LustreError: 209766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6428.799576] LustreError: 207781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443154 with bad export cookie 10948373811283614067 [ 6428.804209] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6428.806875] LustreError: 207781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6429.243883] Lustre: server umount lustre-MDT0001 complete [ 6440.215827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6447.496498] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6458.027354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6464.027776] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6472.488573] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6478.688542] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6486.472520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6495.427195] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6504.627950] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6510.388299] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6512.902248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6518.150650] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6518.415489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6521.824765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6521.849320] LustreError: Skipped 1 previous similar message [ 6526.534140] Lustre: server umount lustre-OST0000 complete [ 6533.875804] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6538.445819] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6567.359968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6573.952828] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6581.449909] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6585.455577] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6587.174521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6587.182156] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6587.182206] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6592.481099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6592.484713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6592.491575] LustreError: Skipped 1 previous similar message [ 6592.492466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6592.492471] Lustre: Skipped 6 previous similar messages [ 6592.542719] Lustre: Skipped 5 previous similar messages [ 6595.413785] Lustre: server umount lustre-OST0000 complete [ 6605.393053] Lustre: server umount lustre-MDT0000 complete [ 6607.846934] LustreError: 213125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6607.857959] LustreError: 213125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6609.622457] LustreError: 211958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443335 with bad export cookie 10948373811283614949 [ 6609.628892] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6609.632445] LustreError: 211958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6609.920310] Lustre: server umount lustre-MDT0001 complete [ 6619.007977] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:49:03 (1788443343) [ 6626.742626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.302881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6627.305369] Lustre: Skipped 6 previous similar messages [ 6631.731206] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6639.796327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6644.057666] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6650.332985] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6658.529720] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6668.316809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6676.221959] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6683.813704] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6688.991432] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6689.178651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6694.566455] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6694.723678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6703.999914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6704.206533] Lustre: lustre-OST0001: new disk, initializing [ 6704.209077] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6704.214588] Lustre: 221328:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6712.012306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6712.027060] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6712.218074] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6712.873635] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6724.494585] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6731.163350] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6731.482368] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6736.988754] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6737.226839] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6743.008613] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6743.018330] LustreError: Skipped 2 previous similar messages [ 6745.449428] Lustre: server umount lustre-OST0001 complete [ 6753.126968] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6753.410914] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6758.265234] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6758.571833] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6789.457703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6795.254352] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6795.292894] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6795.322026] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6795.322038] Lustre: Skipped 1 previous similar message [ 6798.017403] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6806.204974] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6807.729118] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6815.620757] Lustre: server umount lustre-OST0001 complete [ 6848.991154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6849.478077] Lustre: server umount lustre-OST0000 complete [ 6855.141421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6855.178867] Lustre: Skipped 8 previous similar messages [ 6855.190143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6855.196861] Lustre: Skipped 14 previous similar messages [ 6860.077584] Lustre: server umount lustre-MDT0000 complete [ 6864.362578] LustreError: 219801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443590 with bad export cookie 10948373811283615915 [ 6864.376507] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6864.380176] LustreError: 219801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6871.493695] Lustre: server umount lustre-MDT0001 complete [ 6901.673853] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:53:45 (1788443625) [ 6909.905904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.277673] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6910.314960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6916.482757] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6926.184258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6926.256516] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6926.291417] Lustre: 226580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6931.405639] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6940.193663] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6947.048380] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6954.110147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6954.325259] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6960.935481] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6968.734612] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6980.511723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6980.592072] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6988.599772] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 6996.208161] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6998.059436] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7002.678180] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7002.827328] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7008.267879] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7008.444471] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7014.766927] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90c704d22800.ost_server_uuid 50 [ 7016.464461] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90c704d22800.ost_server_uuid in IDLE state after 0 sec [ 7017.376260] Lustre: Permanently deactivating lustre-OST0001 [ 7017.396783] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7021.708464] Lustre: Permanently reactivating lustre-OST0001 [ 7021.763457] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7021.770461] Lustre: Skipped 2 previous similar messages [ 7026.670890] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7026.691285] LustreError: Skipped 2 previous similar messages [ 7032.673720] Lustre: server umount lustre-OST0001 complete [ 7036.899304] LustreError: 228429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7036.930679] LustreError: 228429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 44 previous similar messages [ 7042.850794] Lustre: server umount lustre-OST0000 complete [ 7053.660034] Lustre: server umount lustre-MDT0000 complete [ 7057.747584] LustreError: 226572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443783 with bad export cookie 10948373811283617203 [ 7057.754990] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7057.760667] LustreError: 226572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7058.221775] Lustre: server umount lustre-MDT0001 complete [ 7082.525144] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:56:47 (1788443807) [ 7089.877802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7090.092135] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7090.115319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7090.120934] Lustre: Skipped 2 previous similar messages [ 7094.855465] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7103.427241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7103.529597] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7103.571524] Lustre: 233276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7108.482580] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7116.004789] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7121.909537] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7127.980106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7128.215097] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7128.237500] Lustre: Permanently deactivating lustre-OST0000 [ 7128.254230] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7135.261199] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7138.403532] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7142.957348] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7150.030198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7150.091599] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7150.281230] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7150.285277] Lustre: Skipped 11 previous similar messages [ 7156.556455] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7158.110771] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7164.667323] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7166.538759] Lustre: Permanently reactivating lustre-OST0000 [ 7166.554249] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7166.559249] Lustre: Skipped 2 previous similar messages [ 7168.017906] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7168.033205] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7168.047082] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7168.057578] Lustre: Skipped 1 previous similar message [ 7168.114613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7168.146957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7185.918864] Lustre: server umount lustre-OST0001 complete [ 7196.476600] Lustre: server umount lustre-OST0000 complete [ 7206.537724] Lustre: server umount lustre-MDT0000 complete [ 7210.222752] LustreError: 233269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443936 with bad export cookie 10948373811283618876 [ 7210.243165] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7210.488753] Lustre: server umount lustre-MDT0001 complete [ 7223.109481] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 7226.533197] Key type lgssc unregistered [ 7226.934808] LNet: 238155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7226.945192] LNetError: 238155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7228.007360] LNet: Removed LNI 192.168.201.136@tcp [ 7228.818445] Key type .llcrypt unregistered [ 7228.819823] Key type ._llcrypt unregistered [ 7254.905914] Key type ._llcrypt registered [ 7254.909467] Key type .llcrypt registered [ 7255.017173] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7256.241432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7256.269521] alg: No test for adler32 (adler32-zlib) [ 7257.571728] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7257.805931] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 7259.454431] Key type lgssc registered [ 7260.770345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7272.381928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7272.403987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7274.134843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7279.523195] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7284.705384] LustreError: 239619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7284.744650] LustreError: 239619:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7288.470350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7288.562150] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7288.582541] Lustre: 239609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7288.590287] Lustre: Permanently deactivating lustre-MDT0001 [ 7288.613435] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7288.987080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7293.563926] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7296.584397] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7302.121569] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7308.611762] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7315.503960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7315.901224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7318.970731] LustreError: 241461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7319.009751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7323.541898] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7324.134122] LustreError: 241462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7329.256327] LustreError: 241462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7333.283722] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7334.377335] LustreError: 241463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7334.423304] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7342.773656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7343.330806] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7348.720081] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7348.745069] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7351.197457] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7360.242091] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7364.635414] Lustre: Permanently reactivating lustre-MDT0001 [ 7364.645079] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7364.647611] Lustre: Skipped 2 previous similar messages [ 7366.977367] Lustre: lustre-MDT0001: Client a508823a-75cc-4b2a-8aff-7580d7184f36 (at 192.168.201.36@tcp) reconnecting [ 7371.001055] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7371.034119] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7371.050368] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7371.076187] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7374.524203] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7376.708879] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7384.669569] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7384.997719] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7390.570082] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7390.878158] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7394.177217] Lustre: Permanently deactivating lustre-MDT0001 [ 7394.190914] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7394.193762] Lustre: Skipped 2 previous similar messages [ 7402.063655] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7411.171699] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7411.175889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7411.195035] Lustre: Skipped 1 previous similar message [ 7411.220339] Lustre: Skipped 3 previous similar messages [ 7416.294626] LustreError: 241480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7416.317981] LustreError: 241480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 7416.374758] Lustre: server umount lustre-MDT0000 complete [ 7420.709524] LustreError: 239601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444146 with bad export cookie 44260781529557526 [ 7420.716248] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7420.732151] LustreError: 239601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7421.307851] Lustre: server umount lustre-MDT0001 complete [ 7427.981725] Lustre: server umount lustre-OST0000 complete [ 7439.588377] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:02:44 (1788444164) [ 7449.801078] Lustre: server umount lustre-OST0001 complete [ 7456.376387] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 7470.315793] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7505.685119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.819538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7505.825058] Lustre: Skipped 2 previous similar messages [ 7505.837159] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7505.888362] Lustre: lustre-MDT0000: new disk, initializing [ 7505.988188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7506.001616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7512.362626] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7526.366341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7526.590882] Lustre: 247892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7526.660103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7526.664312] Lustre: Skipped 1 previous similar message [ 7526.778199] Lustre: lustre-MDT0001: new disk, initializing [ 7526.907192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7526.965530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7526.968366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7532.197501] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7542.084735] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7547.596198] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7554.358854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7554.592247] Lustre: lustre-OST0000: new disk, initializing [ 7554.596063] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7554.602049] Lustre: 249860:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7554.674402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7560.283456] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7562.324748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7562.329103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7562.411865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7570.989878] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7573.521459] LustreError: 247899:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7583.537517] LustreError: 247899:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7583.594868] LustreError: 250692:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7584.474106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7584.766768] Lustre: lustre-OST0001: new disk, initializing [ 7584.775744] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7584.790368] Lustre: 250975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7591.797268] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7592.589987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7592.608906] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7592.779996] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7593.608111] LustreError: 250692:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7602.399545] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7604.064387] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7608.289190] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7608.295624] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7608.301314] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7608.308977] Lustre: Skipped 3 previous similar messages [ 7608.329519] Lustre: Skipped 1 previous similar message [ 7613.414981] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7613.432992] Lustre: Skipped 1 previous similar message [ 7618.537813] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7618.544843] Lustre: Skipped 1 previous similar message [ 7620.063476] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7620.168170] Lustre: server umount lustre-OST0001 complete [ 7623.659498] LustreError: 249845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7623.697529] LustreError: 249845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7639.020046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7639.024791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7639.043383] Lustre: Skipped 1 previous similar message [ 7639.059691] Lustre: Skipped 1 previous similar message [ 7643.860983] Lustre: server umount lustre-OST0000 complete [ 7649.249718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7649.272494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7649.289980] Lustre: Skipped 1 previous similar message [ 7649.318226] Lustre: Skipped 1 previous similar message [ 7654.159755] Lustre: server umount lustre-MDT0000 complete [ 7657.965917] LustreError: 247884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444384 with bad export cookie 44260781529560347 [ 7657.982088] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7672.287138] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7672.720596] Lustre: server umount lustre-MDT0001 complete [ 7685.048829] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 7687.802371] Key type lgssc unregistered [ 7688.039909] LNet: 253012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7688.047248] LNetError: 253012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7689.078018] LNet: Removed LNI 192.168.201.136@tcp [ 7690.110731] Key type .llcrypt unregistered [ 7690.113114] Key type ._llcrypt unregistered [ 7733.993748] Key type ._llcrypt registered [ 7733.997930] Key type .llcrypt registered [ 7734.141616] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 7735.954181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7736.059467] alg: No test for adler32 (adler32-zlib) [ 7737.277516] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7737.599972] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 7739.399168] Key type lgssc registered [ 7740.801049] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7749.056605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7749.075364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7750.350519] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7750.375626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7750.598755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7755.638737] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7764.935315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7765.136752] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7765.213299] Lustre: 255217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7765.528210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7770.335455] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7778.054995] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7784.040970] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7790.633816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7790.802935] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7791.050324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7797.191786] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7798.033508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7804.614176] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7864.809957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7864.827969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7864.829028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7864.836885] LustreError: Skipped 1 previous similar message [ 7864.844280] Lustre: Skipped 1 previous similar message [ 7867.265711] Lustre: server umount lustre-OST0000 complete [ 7871.917180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7886.472679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7887.322635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7896.118773] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 7903.788654] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7905.709289] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7905.722792] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7905.723674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7944.677284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7944.680621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7944.691441] Lustre: Skipped 2 previous similar messages [ 7944.708984] Lustre: Skipped 1 previous similar message [ 7949.692904] Lustre: server umount lustre-OST0000 complete [ 7954.913061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7954.922767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7960.033495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7960.050781] Lustre: Skipped 2 previous similar messages [ 7961.209944] Lustre: server umount lustre-MDT0000 complete [ 7965.740860] LustreError: 255208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444691 with bad export cookie 4315729151640604658 [ 7965.760764] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7965.771827] LustreError: 255208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7966.447284] Lustre: server umount lustre-MDT0001 complete [ 7979.129508] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 7982.603345] Key type lgssc unregistered [ 7982.918231] LNet: 260639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7982.923256] LNetError: 260639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7982.945926] LNet: Removed LNI 192.168.201.136@tcp [ 7983.596351] Key type .llcrypt unregistered [ 7983.598297] Key type ._llcrypt unregistered [ 8013.120171] Key type ._llcrypt registered [ 8013.131619] Key type .llcrypt registered [ 8013.375528] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8014.502701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8014.516121] alg: No test for adler32 (adler32-zlib) [ 8016.210414] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8016.535602] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8018.263174] Key type lgssc registered [ 8019.709293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8028.883524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8028.911633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8030.419205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.798996] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8039.905377] LustreError: 262053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8039.915493] LustreError: 262053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8044.161397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8044.693412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8049.426327] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8057.787860] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8064.292950] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8071.572101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8071.998506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8075.061959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8077.992976] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8085.485059] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8116.199432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8116.210205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8116.238057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8120.288976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8120.297773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8120.306915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8121.542699] Lustre: server umount lustre-OST0000 complete [ 8125.408516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8125.418822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8125.424187] Lustre: Skipped 2 previous similar messages [ 8130.536443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8130.560973] Lustre: Skipped 1 previous similar message [ 8131.942736] Lustre: server umount lustre-MDT0000 complete [ 8135.649466] LustreError: 262052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8135.672565] LustreError: 262052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8136.352853] LustreError: 262032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444862 with bad export cookie 13291779566533731298 [ 8136.363467] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8136.377152] LustreError: 262032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8137.270668] Lustre: server umount lustre-MDT0001 complete [ 8152.565390] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8156.710378] Key type lgssc unregistered [ 8156.936274] LNet: 266364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8156.948356] LNetError: 266364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8156.973677] LNet: Removed LNI 192.168.201.136@tcp [ 8157.796326] Key type .llcrypt unregistered [ 8157.798048] Key type ._llcrypt unregistered [ 8177.957322] Key type ._llcrypt registered [ 8177.958559] Key type .llcrypt registered [ 8178.067805] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8180.737774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8180.839285] alg: No test for adler32 (adler32-zlib) [ 8182.266347] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8182.776802] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8184.623448] Key type lgssc registered [ 8186.864356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8197.575474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8197.602910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8199.404705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8204.841492] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8208.870221] LustreError: 267589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8208.887406] LustreError: 267589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8213.990825] LustreError: 267588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8214.678037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8214.917873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8220.407877] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8228.320484] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8235.394643] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8243.270845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8243.758511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8244.816025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8251.301429] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8260.537515] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8275.118926] Lustre: server umount lustre-OST0000 complete [ 8281.063566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8281.068296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8281.083023] Lustre: Skipped 1 previous similar message [ 8281.102220] Lustre: Skipped 1 previous similar message [ 8286.179635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8286.194273] Lustre: Skipped 1 previous similar message [ 8291.300771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8291.307384] Lustre: Skipped 1 previous similar message [ 8292.319232] Lustre: 266897:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445002/real 1788445002] req@00000000aafff388 x1875320407880960/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788445018 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8292.355646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8294.368374] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8294.568611] Lustre: server umount lustre-MDT0000 complete [ 8297.440601] LustreError: 267585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8297.466787] LustreError: 267585:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8297.511244] Lustre: 266897:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445007/real 1788445007] req@000000007027fcd2 x1875320407883264/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8299.337607] LustreError: 267570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445025 with bad export cookie 14788845217587198985 [ 8299.345775] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8299.356469] LustreError: 267570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8300.018742] Lustre: server umount lustre-MDT0001 complete [ 8311.569597] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8314.726125] Key type lgssc unregistered [ 8314.984807] LNet: 271266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8314.998649] LNetError: 271266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8315.023858] LNet: Removed LNI 192.168.201.136@tcp [ 8316.121818] Key type .llcrypt unregistered [ 8316.124989] Key type ._llcrypt unregistered [ 8347.435119] Key type ._llcrypt registered [ 8347.437362] Key type .llcrypt registered [ 8347.539835] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8348.402922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8348.442416] alg: No test for adler32 (adler32-zlib) [ 8349.629747] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8349.870483] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8351.607773] Key type lgssc registered [ 8353.253562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8362.728098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8362.756118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.461356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8370.214989] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8374.755931] LustreError: 272679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8374.794440] LustreError: 272679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8379.877234] LustreError: 272680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8380.309576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.744093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8387.145398] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8396.285663] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8403.521168] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8411.731917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8412.144609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8419.260706] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8428.144512] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8434.679704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8460.260828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8460.266366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8460.289871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8460.319784] Lustre: Skipped 1 previous similar message [ 8460.334136] Lustre: Skipped 1 previous similar message [ 8462.698707] Lustre: server umount lustre-OST0000 complete [ 8467.937030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8467.953260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.980780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8470.497262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8470.498056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8473.579629] Lustre: server umount lustre-MDT0000 complete [ 8475.626794] LustreError: 274976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8475.660628] LustreError: 274976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8477.807427] LustreError: 272662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445203 with bad export cookie 4059831623655337153 [ 8477.821359] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8478.184515] Lustre: server umount lustre-MDT0001 complete [ 8490.416974] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8493.491796] Key type lgssc unregistered [ 8493.893741] LNet: 276987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8493.910052] LNetError: 276987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8493.928811] LNet: Removed LNI 192.168.201.136@tcp [ 8494.658877] Key type .llcrypt unregistered [ 8494.663589] Key type ._llcrypt unregistered [ 8513.905341] Key type ._llcrypt registered [ 8513.912699] Key type .llcrypt registered [ 8514.024921] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8516.170917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8516.286905] alg: No test for adler32 (adler32-zlib) [ 8517.359444] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8517.617254] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8519.279884] Key type lgssc registered [ 8521.463827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8531.364873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8531.412268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8533.183059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8538.102687] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8543.716827] LustreError: 278220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8543.747387] LustreError: 278220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8548.314711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8548.821163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8555.322173] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8565.358275] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8573.840200] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8584.024326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8584.403567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8591.498679] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8599.956752] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8603.840933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8617.194453] Lustre: server umount lustre-OST0000 complete [ 8624.611217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.623591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8624.635818] Lustre: Skipped 1 previous similar message [ 8629.729237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8629.750684] Lustre: Skipped 2 previous similar messages [ 8630.876121] Lustre: server umount lustre-MDT0000 complete [ 8634.683353] LustreError: 278187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445360 with bad export cookie 12506605091222933154 [ 8634.706917] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8644.378901] Lustre: server umount lustre-MDT0001 complete [ 8656.412993] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8659.095964] Key type lgssc unregistered [ 8659.456903] LNet: 281924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8659.467281] LNetError: 281924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8660.522660] LNet: Removed LNI 192.168.201.136@tcp [ 8661.385910] Key type .llcrypt unregistered [ 8661.388468] Key type ._llcrypt unregistered [ 8674.744369] Key type ._llcrypt registered [ 8674.745990] Key type .llcrypt registered [ 8674.953279] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8694.855775] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 8712.566412] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 8713.630820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8713.677465] alg: No test for adler32 (adler32-zlib) [ 8714.783710] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8714.967515] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 8716.631899] Key type lgssc registered [ 8717.846719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8753.910437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8753.922464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8755.143867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8755.175684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8755.350453] Lustre: lustre-MDT0000: new disk, initializing [ 8755.527904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8755.570026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8760.403721] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8770.705105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8770.910949] Lustre: 285474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8770.987307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8771.002045] Lustre: Skipped 1 previous similar message [ 8771.162491] Lustre: lustre-MDT0001: new disk, initializing [ 8771.271165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8771.354942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8771.366823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8776.716699] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8786.076556] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8791.434460] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8797.531236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8797.841682] Lustre: lustre-OST0000: new disk, initializing [ 8797.844516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8797.848723] Lustre: 287442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8797.934846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8803.703245] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8808.000215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8808.009356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8808.101588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8813.084438] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8818.515885] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8818.797434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8823.319396] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8823.611449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8828.896859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8828.900568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8828.909649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8828.923158] Lustre: Skipped 1 previous similar message [ 8831.495390] Lustre: server umount lustre-OST0000 complete [ 8838.112402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8838.119817] LustreError: Skipped 1 previous similar message [ 8838.127793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8838.151627] Lustre: Skipped 1 previous similar message [ 8838.170343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8839.141049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8840.650198] Lustre: server umount lustre-MDT0000 complete [ 8845.237584] LustreError: 285466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445571 with bad export cookie 13766038141912844808 [ 8845.248426] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8845.627892] Lustre: server umount lustre-MDT0001 complete [ 8859.111919] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:26:22 (1788445582) [ 8869.623988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8870.171660] LustreError: 289855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8870.187675] LustreError: 289855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8870.320392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8875.504348] LustreError: 289856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8876.034535] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8880.616616] LustreError: 289855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8885.735169] LustreError: 289856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8886.678930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8887.255801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8891.669708] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8899.589843] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8905.346206] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8912.106525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8912.528804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8919.130334] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 8927.634248] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8939.489053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8939.506096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8939.530926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8939.548607] Lustre: Skipped 2 previous similar messages [ 8941.537792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8941.547277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8941.556277] Lustre: Skipped 1 previous similar message [ 8944.399373] Lustre: server umount lustre-OST0000 complete [ 8948.705242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8948.720253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8948.736265] Lustre: Skipped 1 previous similar message [ 8948.741923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8954.461464] Lustre: server umount lustre-MDT0000 complete [ 8956.901225] LustreError: 290869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8956.921278] LustreError: 290869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8958.503730] LustreError: 291699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445684 with bad export cookie 13766038141912845690 [ 8958.513814] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8958.900270] Lustre: server umount lustre-MDT0001 complete [ 8970.213628] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 8973.615723] Key type lgssc unregistered [ 8974.003631] LNet: 293392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8974.012891] LNetError: 293392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8974.034690] LNet: Removed LNI 192.168.201.136@tcp [ 8975.206964] Key type .llcrypt unregistered [ 8975.211346] Key type ._llcrypt unregistered [ 9005.945839] Key type ._llcrypt registered [ 9005.949279] Key type .llcrypt registered [ 9006.132157] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 9023.942638] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9025.059296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9025.167110] alg: No test for adler32 (adler32-zlib) [ 9026.164794] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9026.371732] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9028.049135] Key type lgssc registered [ 9029.141529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9075.600707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9075.665308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9076.945702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9076.987312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9077.104823] Lustre: lustre-MDT0000: new disk, initializing [ 9077.211758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9077.244714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9082.120248] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9093.612102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.853986] Lustre: 297217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9093.938708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9093.943362] Lustre: Skipped 1 previous similar message [ 9094.055610] Lustre: lustre-MDT0001: new disk, initializing [ 9094.152572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9094.198940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9094.221595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9099.437642] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9109.544804] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9116.498674] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9124.189098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9124.580365] Lustre: lustre-OST2710: new disk, initializing [ 9124.584790] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9124.595180] Lustre: 299186:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9124.681854] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9131.889093] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9133.644947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9133.663364] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9133.829399] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9142.000972] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9149.888817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9150.090096] Lustre: lustre-OST03e8: new disk, initializing [ 9150.093875] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9150.102308] Lustre: 300250:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9150.218784] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9156.288891] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9157.832302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9157.845881] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9157.986740] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9164.251798] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9171.188878] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9171.326218] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9175.802325] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9176.001353] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9180.501943] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9180.760521] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9186.763142] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9187.038437] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9193.962978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.985705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9193.990323] Lustre: Skipped 2 previous similar messages [ 9194.008014] Lustre: Skipped 2 previous similar messages [ 9194.553980] Lustre: server umount lustre-MDT0000 complete [ 9198.757546] LustreError: 297208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445924 with bad export cookie 5854278955441552827 [ 9198.772331] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9199.079280] Lustre: lustre-MDT0001-lwp-OST2710: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9199.097117] Lustre: Skipped 1 previous similar message [ 9199.278961] Lustre: server umount lustre-MDT0001 complete [ 9205.846830] Lustre: server umount lustre-OST2710 complete [ 9210.446991] Lustre: server umount lustre-OST03e8 complete [ 9231.339078] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [ 9242.823403] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9284.680989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9284.905587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9284.923968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9284.985762] Lustre: lustre-MDT0000: new disk, initializing [ 9285.034197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9285.060979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9289.245281] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9300.632872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9300.831482] Lustre: 305495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9300.897842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9300.905511] Lustre: Skipped 1 previous similar message [ 9301.036415] Lustre: lustre-MDT0001: new disk, initializing [ 9301.174983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9301.218115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9301.225779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9305.719298] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9314.655664] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9320.075722] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9328.220270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9328.555808] Lustre: lustre-OST0000: new disk, initializing [ 9328.563477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9328.572101] Lustre: 307465:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9328.722103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9334.362746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9334.396372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9334.710971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9338.262656] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9351.384773] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9356.581315] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9356.929370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9361.429986] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9361.745388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9365.472385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9365.479409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9365.496737] LustreError: Skipped 1 previous similar message [ 9365.501053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9365.528018] Lustre: Skipped 2 previous similar messages [ 9365.565484] Lustre: Skipped 2 previous similar messages [ 9370.596804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9370.609137] Lustre: Skipped 1 previous similar message [ 9375.715064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9375.730352] Lustre: Skipped 1 previous similar message [ 9378.272988] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9378.540539] Lustre: server umount lustre-OST0000 complete [ 9383.393657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9383.396184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9383.407075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9389.011129] Lustre: server umount lustre-MDT0000 complete [ 9391.078702] LustreError: 306307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9391.089352] LustreError: 306307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9392.823542] LustreError: 305486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446118 with bad export cookie 5854278955441554507 [ 9392.825184] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9392.830531] LustreError: 305486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9393.183239] Lustre: server umount lustre-MDT0001 complete [ 9401.736996] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:35:26 (1788446126) [ 9403.685109] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9405.690282] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:35:30 (1788446130) [ 9409.714805] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9438.369940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.653564] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9438.685854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9438.818838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9443.693839] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9451.956638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9452.021892] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9452.039314] Lustre: 311327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9457.029202] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9480.496635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9480.798515] Lustre: Denying initial registration attempt from nid 192.168.201.136@tcp, specified as failover [ 9480.805890] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9480.817194] LustreError: 312782:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9480.831885] LustreError: 312782:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9480.840798] LustreError: 312782:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9480.928430] Lustre: server umount lustre-OST0000 complete [ 9480.931329] LustreError: 312782:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9493.478787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9493.483017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9493.484091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9493.495105] Lustre: Skipped 2 previous similar messages [ 9493.522683] Lustre: Skipped 3 previous similar messages [ 9495.305673] Lustre: server umount lustre-MDT0000 complete [ 9498.600356] LustreError: 311338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9498.617020] LustreError: 311338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9499.512200] LustreError: 311320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446225 with bad export cookie 5854278955441555389 [ 9499.547983] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9499.926764] Lustre: server umount lustre-MDT0001 complete [ 9511.655743] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9514.289942] Key type lgssc unregistered [ 9514.546884] LNet: 313901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9514.556558] LNetError: 313901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9514.579835] LNet: Removed LNI 192.168.201.136@tcp [ 9515.367481] Key type .llcrypt unregistered [ 9515.369971] Key type ._llcrypt unregistered [ 9527.449487] Key type ._llcrypt registered [ 9527.454138] Key type .llcrypt registered [ 9527.622328] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9528.723161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9528.832163] alg: No test for adler32 (adler32-zlib) [ 9529.836421] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9530.109821] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9531.871171] Key type lgssc registered [ 9532.924438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9580.732461] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9590.240773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9590.273396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9591.641920] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9591.669785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9591.887593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9596.448877] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9605.012934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9605.096235] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9605.134674] Lustre: 316662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9605.372481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9610.792667] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9619.140635] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9625.660773] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9646.759268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9646.998516] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9647.243199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9654.079676] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9661.542398] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9667.552383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9667.564927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9667.566668] LustreError: Skipped 1 previous similar message [ 9667.567752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9667.600969] Lustre: Skipped 1 previous similar message [ 9669.521357] Lustre: server umount lustre-OST0000 complete [ 9677.281539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9677.289231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9677.304203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9677.312262] Lustre: Skipped 1 previous similar message [ 9679.529764] Lustre: server umount lustre-MDT0000 complete [ 9683.058342] LustreError: 316652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446409 with bad export cookie 13916810229858379100 [ 9683.072335] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9683.075916] LustreError: 316652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9683.382913] Lustre: server umount lustre-MDT0001 complete [ 9695.077361] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9698.409803] Key type lgssc unregistered [ 9698.806596] LNet: 320234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9698.816848] LNetError: 320234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9698.835815] LNet: Removed LNI 192.168.201.136@tcp [ 9699.836533] Key type .llcrypt unregistered [ 9699.842385] Key type ._llcrypt unregistered [ 9726.035139] Key type ._llcrypt registered [ 9726.037564] Key type .llcrypt registered [ 9726.142131] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9727.667048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9727.728521] alg: No test for adler32 (adler32-zlib) [ 9728.801772] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9729.115659] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9730.811474] Key type lgssc registered [ 9732.183153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9739.292807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9739.312188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9740.830182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9745.409553] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9749.986661] LustreError: 321645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9749.995146] LustreError: 321645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9753.717576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9754.162919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9758.705522] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9766.826495] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9773.161842] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9782.832997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9783.232882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9790.169055] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9797.802879] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9815.014982] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9815.018646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9815.038373] Lustre: Skipped 1 previous similar message [ 9818.204632] Lustre: server umount lustre-MDT0000 complete [ 9820.131304] LustreError: 323048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9820.154778] LustreError: 323048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9822.383815] LustreError: 321628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446548 with bad export cookie 14635859310080990727 [ 9822.395124] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9825.251838] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9825.251840] LustreError: 321640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9825.251848] Lustre: Skipped 1 previous similar message [ 9825.255312] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9825.282065] LustreError: 321640:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9825.289066] Lustre: Skipped 3 previous similar messages [ 9827.296639] LustreError: 321642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9827.297134] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9827.325188] LustreError: 321642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9827.345961] Lustre: Skipped 1 previous similar message [ 9828.983358] Lustre: server umount lustre-MDT0001 complete [ 9835.761357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9844.277503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9847.777899] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcb1d047f97f8bca7 [ 9847.794039] Lustre: MGC192.168.201.136@tcp: Connection restored to 0@lo (at 0@lo) [ 9848.118273] LustreError: 323479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9848.189670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9848.251249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9852.837405] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9857.378728] LustreError: 325260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9857.391134] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9857.417609] LustreError: 325260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9862.825879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9863.284342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9867.975534] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9868.777075] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9875.698845] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9877.363938] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9883.667699] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9885.115104] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9893.876441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9893.884944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9893.899126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9895.982931] Lustre: server umount lustre-OST0000 complete [ 9904.100651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9904.104800] LustreError: Skipped 1 previous similar message [ 9904.109618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9904.116923] Lustre: Skipped 1 previous similar message [ 9904.120222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9904.127908] Lustre: Skipped 2 previous similar messages [ 9905.945832] Lustre: server umount lustre-MDT0000 complete [ 9909.730138] LustreError: 325265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9909.747869] LustreError: 325265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9910.211733] LustreError: 327182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446636 with bad export cookie 14635859310080998567 [ 9910.227936] LustreError: 327182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9910.242562] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9910.710911] Lustre: server umount lustre-MDT0001 complete [ 9921.139950] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [ 9923.952438] Key type lgssc unregistered [ 9924.269691] LNet: 327933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9924.279646] LNetError: 327933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9924.303890] LNet: Removed LNI 192.168.201.136@tcp [ 9925.034301] Key type .llcrypt unregistered [ 9925.036487] Key type ._llcrypt unregistered [ 9952.767416] Key type ._llcrypt registered [ 9952.770529] Key type .llcrypt registered [ 9952.923649] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [ 9954.231382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9954.255265] alg: No test for adler32 (adler32-zlib) [ 9955.303605] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9955.576573] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [ 9957.311236] Key type lgssc registered [ 9958.323064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9966.011832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9966.025155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9967.811236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9973.030377] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9973.224841] LustreError: 329349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9973.259536] LustreError: 329349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9977.378190] LustreError: 329350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9981.712231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9982.186507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9987.536679] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [ 9994.405323] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9999.839931] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10003.308189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10003.326539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10003.333606] Lustre: Skipped 1 previous similar message [10007.463589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10007.477487] Lustre: Skipped 1 previous similar message [10009.591240] Lustre: server umount lustre-MDT0000 complete [10012.644828] LustreError: 330588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10012.663604] LustreError: 330588:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10013.125528] LustreError: 329328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446739 with bad export cookie 3351966952214437422 [10013.136914] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10013.148073] LustreError: 329328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10013.686462] Lustre: server umount lustre-MDT0001 complete [10028.050258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.432529] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10028.458760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10028.763346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10033.359485] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10043.174468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10043.303260] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10043.338314] Lustre: 331934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10043.646501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10048.612076] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10056.067575] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10061.875919] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10072.238109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10072.395142] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10072.401745] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10072.406362] LustreError: 333819:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10072.413084] LustreError: 333819:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10072.419327] LustreError: 333819:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10072.501624] Lustre: server umount lustre-OST0000 complete [10072.503515] LustreError: 333819:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10082.933352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10083.262563] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10083.545214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10090.520700] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10093.697221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10097.884829] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10106.033629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10106.127368] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10106.238827] Lustre: lustre-OST0001: new disk, initializing [10106.244455] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10106.262119] Lustre: 335360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10106.394810] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10113.164579] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10115.298632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10115.304202] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10115.417074] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10121.106730] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10125.800315] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10125.803765] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10125.814340] Lustre: Skipped 2 previous similar messages [10125.815701] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10125.822491] Lustre: Skipped 1 previous similar message [10129.920995] Lustre: server umount lustre-OST0001 complete [10130.931529] LustreError: 334285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10130.945708] LustreError: 334285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10134.496226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10134.500746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10134.527560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10136.037886] LustreError: 334285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10136.047548] LustreError: 334285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10141.156339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10141.164125] Lustre: Skipped 4 previous similar messages [10146.277885] LustreError: 334287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10146.297310] LustreError: 334287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10147.295496] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10147.474766] Lustre: server umount lustre-OST0000 complete [10156.513166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10156.515068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10156.520606] Lustre: Skipped 1 previous similar message [10156.531422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10156.538311] LustreError: Skipped 1 previous similar message [10156.554693] Lustre: Skipped 3 previous similar messages [10157.874402] Lustre: server umount lustre-MDT0000 complete [10161.096067] LustreError: 332674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446887 with bad export cookie 3351966952214437975 [10161.097268] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10161.105096] LustreError: 332674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10161.486323] Lustre: server umount lustre-MDT0001 complete [10172.492761] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10175.827038] Key type lgssc unregistered [10176.191135] LNet: 337318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10176.204150] LNetError: 337318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10176.224950] LNet: Removed LNI 192.168.201.136@tcp [10177.239976] Key type .llcrypt unregistered [10177.242427] Key type ._llcrypt unregistered [10245.457226] Key type ._llcrypt registered [10245.459387] Key type .llcrypt registered [10245.583488] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [10258.398747] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10259.686174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10259.720441] alg: No test for adler32 (adler32-zlib) [10260.785933] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10261.023334] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [10262.687289] Key type lgssc registered [10264.163810] Lustre: Echo OBD driver; http://www.lustre.org/ [10297.064259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10297.103261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10298.388487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10298.414297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10298.484707] Lustre: lustre-MDT0000: new disk, initializing [10298.546038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10298.559562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10302.115457] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10311.964641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10312.115478] Lustre: 342227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10312.177754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10312.180489] Lustre: Skipped 1 previous similar message [10312.266630] Lustre: lustre-MDT0001: new disk, initializing [10312.365047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10312.413668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10312.440371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10316.818774] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10325.685658] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10331.130363] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10338.198305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10338.556263] Lustre: lustre-OST0000: new disk, initializing [10338.560860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10338.573898] Lustre: 344194:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10338.731975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10344.492920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10344.511317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10344.654178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10345.169756] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10353.431443] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10358.506941] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10358.707948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10363.930655] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10364.241587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10370.533305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10370.534719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10370.542654] LustreError: Skipped 1 previous similar message [10370.550305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10372.773463] Lustre: server umount lustre-OST0000 complete [10379.233241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10379.251262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10379.274513] Lustre: Skipped 1 previous similar message [10379.288649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10379.298418] Lustre: Skipped 1 previous similar message [10380.769405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10380.771484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10380.816055] Lustre: Skipped 1 previous similar message [10384.485566] Lustre: server umount lustre-MDT0000 complete [10388.352171] LustreError: 342220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447114 with bad export cookie 11109819411706610389 [10388.357096] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10388.365580] LustreError: 342220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10388.597217] Lustre: server umount lustre-MDT0001 complete [10398.803840] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:52:03 (1788447123) [10407.021125] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:52:11 (1788447131) [10414.354884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10414.723224] LustreError: 346948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10414.747653] LustreError: 346948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10414.795112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10420.202785] LustreError: 346949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10420.254321] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10424.291274] LustreError: 346948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10429.432449] LustreError: 346949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10430.756985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10431.146077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10436.068727] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10444.807478] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10451.681698] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10457.218707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10457.546045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10463.248960] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10470.106721] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10481.750573] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10483.370979] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10485.173507] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10486.640870] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10488.390676] Lustre: Failing over lustre-MDT0000 [10488.624731] Lustre: server umount lustre-MDT0000 complete [10491.368160] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10491.375250] LustreError: 348792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10491.393863] Lustre: Skipped 1 previous similar message [10501.601490] LustreError: 346945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10501.613500] LustreError: 346945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10507.294648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10507.445077] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10507.730653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10507.799669] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10508.563228] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10512.225127] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10512.892535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10512.936100] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10513.005758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10521.884707] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10523.576802] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10527.168581] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10529.139318] Lustre: Failing over lustre-MDT0000 [10529.509419] Lustre: server umount lustre-MDT0000 complete [10533.345808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10533.350936] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10533.372289] LustreError: 346949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10533.379977] Lustre: Skipped 2 previous similar messages [10533.398274] LustreError: 346949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10549.728295] Lustre: 340450:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447259/real 1788447259] req@00000000275382c5 x1875322587931648/t0(0) o400->MGC192.168.201.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788447275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10549.766510] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10550.546702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10558.965492] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9a2e01e481830d6a [10558.994403] Lustre: MGC192.168.201.136@tcp: Connection restored to 0@lo (at 0@lo) [10558.999801] Lustre: Skipped 2 previous similar messages [10559.224872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10559.269794] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10560.788271] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10563.880592] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10564.590458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10564.628917] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10564.667186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10573.323559] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10574.958981] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10585.057420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10585.062083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10585.071609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10585.096557] Lustre: Skipped 3 previous similar messages [10587.024705] Lustre: server umount lustre-MDT0000 complete [10590.178153] LustreError: 346948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10590.203244] LustreError: 346948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [10590.331260] LustreError: 346928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447316 with bad export cookie 11109819411706613098 [10590.337248] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10590.348617] LustreError: 346928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10590.761830] Lustre: server umount lustre-MDT0001 complete [10605.286855] Lustre: server umount lustre-OST0000 complete [10621.074862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10626.337173] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10634.781529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10635.099384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10635.117551] Lustre: Skipped 1 previous similar message [10639.430844] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10646.545209] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10652.389295] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10659.790779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10666.903813] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10667.462121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10674.525946] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10681.633752] Lustre: server umount lustre-OST0000 complete [10686.433174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10686.445813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10686.478895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10686.490877] Lustre: Skipped 2 previous similar messages [10695.219165] Lustre: server umount lustre-MDT0000 complete [10697.188200] Lustre: 340449:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447407/real 1788447407] req@000000007d5c9fc7 x1875322587984000/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788447423 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10697.215460] LustreError: 354553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10697.240556] LustreError: 354553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10698.531850] LustreError: 353293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447424 with bad export cookie 11109819411706613854 [10698.539727] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10698.546874] LustreError: 353293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10698.823049] Lustre: server umount lustre-MDT0001 complete [10710.153196] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [10713.063319] Key type lgssc unregistered [10713.440819] LNet: 356841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10713.457808] LNetError: 356841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10713.475761] LNet: Removed LNI 192.168.201.136@tcp [10714.682827] Key type .llcrypt unregistered [10714.688348] Key type ._llcrypt unregistered [10744.636553] Key type ._llcrypt registered [10744.641564] Key type .llcrypt registered [10744.723664] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [10745.919936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10745.965093] alg: No test for adler32 (adler32-zlib) [10746.998425] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10747.216238] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [10748.938300] Key type lgssc registered [10750.018677] Lustre: Echo OBD driver; http://www.lustre.org/ [10759.014279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10759.047452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10760.632602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10765.445527] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10769.898224] LustreError: 358354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10769.923858] LustreError: 358354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10774.132047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10774.537985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10778.611549] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10784.616520] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10789.588579] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10797.190253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10797.422883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10805.039169] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10805.616466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10812.795952] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10825.698312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10825.713754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10825.800234] LustreError: 358354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10826.214820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10826.239627] Lustre: Skipped 1 previous similar message [10827.839931] Lustre: server umount lustre-MDT0000 complete [10831.340057] LustreError: 358349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10831.349038] LustreError: 358349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10832.044618] LustreError: 358334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447558 with bad export cookie 3701559624436802571 [10832.048698] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10832.055547] LustreError: 358334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10832.421460] Lustre: server umount lustre-MDT0001 complete [10846.625510] Lustre: server umount lustre-OST0000 complete [10860.280559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10860.793116] LustreError: 362062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10860.908349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10865.543203] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10866.146939] LustreError: 362063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10875.367318] LustreError: 362063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10875.407517] LustreError: 362063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10875.809819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10876.169676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10880.947090] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10887.736170] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10893.981241] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10900.328723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10900.789187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10904.884327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10907.464986] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10915.396584] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10927.588741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10927.614777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10927.652241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10930.660946] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10930.662176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10930.677243] Lustre: Skipped 1 previous similar message [10930.698246] Lustre: Skipped 2 previous similar messages [10933.474655] Lustre: server umount lustre-MDT0000 complete [10935.789330] LustreError: 362062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.804655] LustreError: 362062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10936.894425] LustreError: 362044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447663 with bad export cookie 3701559624436803677 [10936.904906] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10936.906462] LustreError: 362044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10937.273469] Lustre: server umount lustre-MDT0001 complete [10951.646049] Lustre: server umount lustre-OST0000 complete [10966.792355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10967.260904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10971.591248] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10972.641030] LustreError: 365989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10972.666574] LustreError: 365989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10979.541623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10984.379839] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [10991.737896] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10997.202620] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11003.764528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11004.271586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11004.282036] Lustre: Skipped 1 previous similar message [11011.199475] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11012.496772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11018.634624] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11022.816729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11022.822554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11022.827493] Lustre: Skipped 1 previous similar message [11027.942703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11027.947221] Lustre: Skipped 2 previous similar messages [11029.041090] Lustre: server umount lustre-OST0000 complete [11033.058145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11033.059481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11033.073653] Lustre: Skipped 1 previous similar message [11033.085992] Lustre: Skipped 1 previous similar message [11038.826966] Lustre: server umount lustre-MDT0000 complete [11042.006717] LustreError: 365970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447768 with bad export cookie 3701559624436804699 [11042.012788] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11042.021112] LustreError: 365970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11042.327560] Lustre: server umount lustre-MDT0001 complete [11051.272743] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11053.812420] Key type lgssc unregistered [11054.168885] LNet: 369518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11054.180752] LNetError: 369518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11055.207515] LNet: Removed LNI 192.168.201.136@tcp [11056.055194] Key type .llcrypt unregistered [11056.056606] Key type ._llcrypt unregistered [11080.193606] Key type ._llcrypt registered [11080.199621] Key type .llcrypt registered [11080.351769] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11081.657920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11081.764367] alg: No test for adler32 (adler32-zlib) [11082.937711] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11083.204285] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11084.970955] Key type lgssc registered [11086.423551] Lustre: Echo OBD driver; http://www.lustre.org/ [11095.418987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11095.458282] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11095.472483] LustreError: 370955:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11095.493844] LustreError: 370955:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11095.505712] LustreError: 370955:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11095.515958] LustreError: 370955:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11095.521251] LustreError: 370955:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11103.757030] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11103.761366] LustreError: 371352:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11103.775384] LustreError: 371352:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11103.783498] LustreError: 371352:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11103.793415] LustreError: 371352:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11103.803325] LustreError: 371352:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11123.371418] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11126.364802] Key type lgssc unregistered [11126.681293] LNet: 372355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11126.687567] LNetError: 372355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11126.705363] LNet: Removed LNI 192.168.201.136@tcp [11127.396035] Key type .llcrypt unregistered [11127.401994] Key type ._llcrypt unregistered [11145.209391] Key type ._llcrypt registered [11145.212382] Key type .llcrypt registered [11145.346945] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [11159.910210] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11160.866599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11160.907907] alg: No test for adler32 (adler32-zlib) [11161.877505] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11162.034672] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11163.663202] Key type lgssc registered [11164.778613] Lustre: Echo OBD driver; http://www.lustre.org/ [11197.853436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11197.880675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11199.077786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11199.105108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11199.179156] Lustre: lustre-MDT0000: new disk, initializing [11199.253989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11199.262622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11202.979256] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11213.394448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11213.571765] Lustre: 375504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11213.654873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11213.662754] Lustre: Skipped 1 previous similar message [11213.921124] Lustre: lustre-MDT0001: new disk, initializing [11214.157739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11214.370257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11214.405902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11219.533907] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11229.956505] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11235.214798] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11242.227348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11242.561955] Lustre: lustre-OST0000: new disk, initializing [11242.567573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11242.574512] Lustre: 377475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11242.670117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11248.720959] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11252.301000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11252.306785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11252.400902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11257.733140] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11262.089188] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11262.303349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11266.264561] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11266.408155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11268.076260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11268.077266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11268.081845] LustreError: Skipped 1 previous similar message [11268.083969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11268.097509] Lustre: Skipped 1 previous similar message [11273.188680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11273.193333] Lustre: Skipped 2 previous similar messages [11278.304966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11278.308211] Lustre: Skipped 1 previous similar message [11282.399310] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11282.558692] Lustre: server umount lustre-OST0000 complete [11288.546323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11288.573953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11288.586378] Lustre: Skipped 1 previous similar message [11293.664390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11293.677326] Lustre: Skipped 1 previous similar message [11301.343705] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11301.593592] Lustre: server umount lustre-MDT0000 complete [11305.023644] LustreError: 375496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448031 with bad export cookie 13823859150602381511 [11305.034736] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11305.035517] LustreError: 375496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11305.334744] Lustre: server umount lustre-MDT0001 complete [11312.697871] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11314.099993] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:07:19 (1788448039) [11321.295104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11321.596808] LustreError: 379938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.607377] LustreError: 379938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11321.651301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11325.621604] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11326.970661] LustreError: 379939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11331.107124] LustreError: 379938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11334.369633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11334.599609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11339.262719] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11345.528623] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11350.520661] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11355.078802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11355.418405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11362.293277] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11370.043406] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11377.191763] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11377.330321] Lustre: lustre-OST0001: new disk, initializing [11377.333188] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11377.338623] Lustre: 382774:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11377.431836] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11383.609976] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11386.968100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11386.978736] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11387.087272] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11392.344306] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11397.606361] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11397.609033] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11397.613682] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11397.613689] Lustre: Skipped 2 previous similar messages [11397.633656] Lustre: Skipped 2 previous similar messages [11402.881714] Lustre: server umount lustre-OST0001 complete [11405.087576] LustreError: 381759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11412.973285] LustreError: 381760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11412.986657] LustreError: 381760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11418.086233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11418.088836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11418.089565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11418.089570] Lustre: Skipped 4 previous similar messages [11418.092093] LustreError: Skipped 1 previous similar message [11418.114451] Lustre: Skipped 1 previous similar message [11423.209470] LustreError: 381757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11423.230396] LustreError: 381757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11424.197989] Lustre: server umount lustre-OST0000 complete [11428.323363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11442.144646] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11442.303395] Lustre: server umount lustre-MDT0000 complete [11443.684858] LustreError: 380666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11443.726495] LustreError: 380666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11446.029306] LustreError: 379918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448172 with bad export cookie 13823859150602382393 [11446.032723] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11446.044304] LustreError: 379918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11452.335720] Lustre: server umount lustre-MDT0001 complete [11460.951561] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11463.819238] Key type lgssc unregistered [11464.055837] LNet: 384765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11464.064539] LNetError: 384765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11464.079289] LNet: Removed LNI 192.168.201.136@tcp [11464.985251] Key type .llcrypt unregistered [11464.987941] Key type ._llcrypt unregistered [11514.348630] Key type ._llcrypt registered [11514.350501] Key type .llcrypt registered [11514.469480] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11516.105254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11516.176836] alg: No test for adler32 (adler32-zlib) [11517.492926] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11517.920443] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11519.599211] Key type lgssc registered [11520.967942] Lustre: Echo OBD driver; http://www.lustre.org/ [11530.280589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11530.302495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11531.656169] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11531.712750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11531.955453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11537.856578] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11546.972780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11547.119203] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11547.160820] Lustre: 387572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11547.562031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11551.729978] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11559.841463] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11567.326316] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11571.173059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11571.193158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11571.203714] Lustre: Skipped 1 previous similar message [11576.121911] Lustre: server umount lustre-MDT0000 complete [11580.619657] LustreError: 387562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448306 with bad export cookie 12645111935880241946 [11580.633469] LustreError: 387562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11580.634427] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11581.040659] Lustre: server umount lustre-MDT0001 complete [11591.246935] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:11:55 (1788448315) [11598.499794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11599.053528] LustreError: 390082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11599.072189] LustreError: 390082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11599.133702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11603.942597] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11604.459673] LustreError: 390083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11609.572550] LustreError: 390082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11614.143876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11614.436617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11618.400782] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11624.793362] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11629.830678] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11636.332988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11636.554987] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11636.817465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11643.976070] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11653.107560] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11662.640853] Lustre: Permanently deactivating lustre-OST0000 [11662.667120] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11671.633962] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11673.864536] LustreError: 393119:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11673.873777] LustreError: 393119:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11673.879913] LustreError: 393119:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11678.324600] LustreError: 393167:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11678.332721] LustreError: 393167:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11683.279763] Lustre: server umount lustre-OST0000 complete [11686.509719] LustreError: 393416:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11686.517741] LustreError: 393416:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11690.979271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11690.995527] Lustre: Skipped 2 previous similar messages [11690.997594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11691.016628] Lustre: Skipped 1 previous similar message [11696.096619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11696.107316] Lustre: Skipped 1 previous similar message [11701.217046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11701.221586] Lustre: Skipped 1 previous similar message [11703.263214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11703.438924] Lustre: server umount lustre-MDT0000 complete [11706.346246] LustreError: 390077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11706.363470] LustreError: 390077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11707.292574] LustreError: 390063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448433 with bad export cookie 12645111935880242527 [11707.302236] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11707.312502] LustreError: 390063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11717.775381] Lustre: server umount lustre-MDT0001 complete [11726.779355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11731.646613] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11734.278980] LustreError: 394577:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11740.490948] Lustre: 394916:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.136@tcp [11742.546834] Lustre: server umount MGS complete [11752.917919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11753.276349] LustreError: 395639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11753.293401] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11753.302277] Lustre: Skipped 1 previous similar message [11753.370573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11758.097908] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11763.688479] LustreError: 395639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11763.705666] LustreError: 395639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11766.867367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11767.093279] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11772.252995] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11779.854658] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11785.525608] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11787.316269] Lustre: Permanently reactivating lustre-OST0000 [11787.329945] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11787.334177] Lustre: Skipped 2 previous similar messages [11817.951144] LustreError: 397491:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11823.974868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11824.236086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11824.244182] Lustre: Skipped 1 previous similar message [11830.480152] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11837.588614] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11843.551933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11843.560359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11843.582248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11849.725828] Lustre: server umount lustre-OST0000 complete [11854.304585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11854.307857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11854.313417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11854.325522] Lustre: Skipped 2 previous similar messages [11854.333554] Lustre: Skipped 4 previous similar messages [11859.432241] LustreError: 398409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11859.450936] LustreError: 398409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11859.524593] Lustre: server umount lustre-MDT0000 complete [11863.157631] LustreError: 397214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448589 with bad export cookie 12645111935880243577 [11863.161265] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11863.173731] LustreError: 397214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11863.484914] Lustre: server umount lustre-MDT0001 complete [11874.840939] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [11877.170188] Key type lgssc unregistered [11877.419784] LNet: 400101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11877.423486] LNetError: 400101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11877.441725] LNet: Removed LNI 192.168.201.136@tcp [11878.272753] Key type .llcrypt unregistered [11878.277300] Key type ._llcrypt unregistered [11894.925812] Key type ._llcrypt registered [11894.927869] Key type .llcrypt registered [11895.013398] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [11910.480519] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [11912.043627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11912.067019] alg: No test for adler32 (adler32-zlib) [11913.168708] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11913.378430] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [11915.056563] Key type lgssc registered [11916.811912] Lustre: Echo OBD driver; http://www.lustre.org/ [11953.869345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11953.884803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11955.280216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11955.338989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11955.474574] Lustre: lustre-MDT0000: new disk, initializing [11955.567883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11955.589807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11960.898553] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11971.730269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11971.825975] Lustre: 403258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11971.873496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11971.876520] Lustre: Skipped 1 previous similar message [11971.925752] Lustre: lustre-MDT0001: new disk, initializing [11971.995077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11972.022563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11972.027446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11976.321259] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [11985.205919] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11991.195394] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11998.353281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11998.634144] Lustre: lustre-OST0000: new disk, initializing [11998.637876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11998.651180] Lustre: 405224:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11998.762031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12004.978121] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12007.050887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12007.063457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12007.154896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12014.519992] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12019.606921] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12019.797065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12024.884793] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12025.115820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12027.874700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12027.878541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12027.885784] LustreError: Skipped 1 previous similar message [12027.897781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12027.915121] Lustre: Skipped 1 previous similar message [12027.937217] Lustre: Skipped 1 previous similar message [12033.008383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12033.014918] Lustre: Skipped 1 previous similar message [12036.064780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12036.078285] Lustre: Skipped 1 previous similar message [12041.186962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12041.189244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [12041.198571] Lustre: Skipped 1 previous similar message [12041.432421] Lustre: server umount lustre-OST0000 complete [12046.319913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12046.327468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12046.338940] Lustre: Skipped 1 previous similar message [12046.358668] Lustre: Skipped 1 previous similar message [12051.363466] Lustre: server umount lustre-MDT0000 complete [12055.212079] LustreError: 403250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448781 with bad export cookie 8023028723677396044 [12055.217350] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12055.220748] LustreError: 403250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12055.508185] Lustre: server umount lustre-MDT0001 complete [12064.493108] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12066.755581] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:19:51 (1788448791) [12073.060096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12073.424839] LustreError: 407684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12073.439903] LustreError: 407684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12073.502756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12078.232583] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12078.563385] LustreError: 407685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12083.687578] LustreError: 407684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12086.838047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12087.290648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12091.775825] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12099.374979] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12105.070459] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12112.148320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12112.526415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12118.725920] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12126.585089] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12128.632834] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12131.365787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12139.499188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12139.511782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12139.513892] LustreError: Skipped 1 previous similar message [12139.514299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12139.553076] Lustre: Skipped 1 previous similar message [12142.835953] Lustre: server umount lustre-OST0000 complete [12148.704484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12148.711087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12152.576940] Lustre: server umount lustre-MDT0000 complete [12154.851037] LustreError: 408412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12154.871687] LustreError: 408412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12156.669402] LustreError: 410659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448882 with bad export cookie 8023028723677396919 [12156.674669] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12156.683985] LustreError: 410659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12157.063420] Lustre: server umount lustre-MDT0001 complete [12168.054532] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12171.342702] Key type lgssc unregistered [12171.597846] LNet: 411409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12171.609739] LNetError: 411409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12172.647360] LNet: Removed LNI 192.168.201.136@tcp [12173.625296] Key type .llcrypt unregistered [12173.627402] Key type ._llcrypt unregistered [12198.100781] Key type ._llcrypt registered [12198.102677] Key type .llcrypt registered [12198.177812] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12198.470473] Key type .llcrypt unregistered [12198.471629] Key type ._llcrypt unregistered [12218.272352] Key type ._llcrypt registered [12218.276288] Key type .llcrypt registered [12218.456393] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12219.473012] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12219.598429] alg: No test for adler32 (adler32-zlib) [12220.703991] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12221.083723] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [12222.863196] Key type lgssc registered [12224.307376] Lustre: Echo OBD driver; http://www.lustre.org/ [12232.124055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12232.135434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12233.719797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12238.224128] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12242.924623] LustreError: 413618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12242.944677] LustreError: 413618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12246.792625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12247.257514] LustreError: 413619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12247.314458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12252.649483] LustreError: 413618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12252.665026] LustreError: 413618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12254.093700] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12257.772766] LustreError: 413619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12257.793264] LustreError: 413619:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12258.814891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12261.446497] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12263.910623] LustreError: 413618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12263.940894] LustreError: 413618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12268.307774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12268.696375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12274.181947] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12283.875055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12283.888470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12283.928761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12284.393338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12289.431371] Lustre: server umount lustre-OST0000 complete [12294.112851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12294.120868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12294.138837] Lustre: Skipped 1 previous similar message [12294.144032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12294.147133] Lustre: Skipped 2 previous similar messages [12299.625856] Lustre: server umount lustre-MDT0000 complete [12299.748812] LustreError: 413615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12299.772315] LustreError: 413615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12303.620232] LustreError: 413600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449029 with bad export cookie 16149281860363546136 [12303.633572] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12303.987530] Lustre: server umount lustre-MDT0001 complete [12313.089700] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12315.468840] Key type lgssc unregistered [12315.881207] LNet: 416766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12315.891627] LNetError: 416766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12315.922921] LNet: Removed LNI 192.168.201.136@tcp [12316.714593] Key type .llcrypt unregistered [12316.718128] Key type ._llcrypt unregistered [12342.602557] Key type ._llcrypt registered [12342.605116] Key type .llcrypt registered [12342.692887] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12343.635381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12343.718460] alg: No test for adler32 (adler32-zlib) [12344.804726] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12345.005426] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [12346.687186] Key type lgssc registered [12348.067456] Lustre: Echo OBD driver; http://www.lustre.org/ [12355.613575] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12355.623954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12371.103418] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12376.223326] LustreError: 418152:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.136@tcp: failed processing log, type 4: rc = -110 [12402.915333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12409.537636] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12416.430904] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12421.510819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12422.027285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12422.093439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12426.344952] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12427.233854] LustreError: 418177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12427.244375] LustreError: 418177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12432.360349] LustreError: 418178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12436.229815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12436.800375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12442.865053] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12450.827631] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12455.621491] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12463.076069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12463.084915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12463.103165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12463.604705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12463.628163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12465.024761] Lustre: server umount lustre-OST0000 complete [12468.708254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12468.715771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12473.850485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12473.855081] Lustre: Skipped 2 previous similar messages [12474.624397] Lustre: server umount lustre-MDT0000 complete [12478.185798] LustreError: 418158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449204 with bad export cookie 9594556197753762422 [12478.192473] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12478.522345] Lustre: server umount lustre-MDT0001 complete [12488.169959] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12490.851296] Key type lgssc unregistered [12491.144724] LNet: 421721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12491.164904] LNetError: 421721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12491.186171] LNet: Removed LNI 192.168.201.136@tcp [12491.999859] Key type .llcrypt unregistered [12492.001903] Key type ._llcrypt unregistered [12520.845747] Key type ._llcrypt registered [12520.847501] Key type .llcrypt registered [12520.911325] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12521.683675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12521.754424] alg: No test for adler32 (adler32-zlib) [12522.816981] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12523.123284] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [12524.863184] Key type lgssc registered [12525.878293] Lustre: Echo OBD driver; http://www.lustre.org/ [12534.711217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12534.737829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12536.377359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12541.544494] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12546.541362] LustreError: 423133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12546.573287] LustreError: 423133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12549.323163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12549.996303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12555.820216] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12564.389772] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12569.474293] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12575.724496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12576.049747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12581.193532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12582.108917] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12589.187734] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12594.344440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12594.346947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12594.367845] Lustre: Skipped 1 previous similar message [12595.507801] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.36@tcp (stopping) [12595.513176] Lustre: Skipped 2 previous similar messages [12598.686802] Lustre: server umount lustre-MDT0000 complete [12599.782785] LustreError: 423130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12599.810322] LustreError: 423130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12600.800739] LustreError: 423113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.36@tcp arrived at 1788449326 with bad export cookie 13305810716756496478 [12602.495073] LustreError: 423115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449328 with bad export cookie 13305810716756495589 [12602.498285] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12602.509402] LustreError: 423115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12604.896271] LustreError: 423660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12604.896752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12604.906896] LustreError: 423660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12604.923663] Lustre: Skipped 1 previous similar message [12604.927279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12607.909224] LustreError: 423134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12607.912302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12607.932913] LustreError: 423134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12608.855598] Lustre: server umount lustre-OST0000 complete [12612.012240] LustreError: 425712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12612.022570] LustreError: 425712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12616.927351] LustreError: 423866:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000526f0cc3 x1875324959751040/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12617.228129] Lustre: server umount lustre-MDT0001 complete [12627.288150] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12630.591536] Key type lgssc unregistered [12630.888514] LNet: 426818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12630.901163] LNetError: 426818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12630.922777] LNet: Removed LNI 192.168.201.136@tcp [12632.331171] Key type .llcrypt unregistered [12632.338804] Key type ._llcrypt unregistered [12658.190329] Key type ._llcrypt registered [12658.193421] Key type .llcrypt registered [12658.329621] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12659.539162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12659.742378] alg: No test for adler32 (adler32-zlib) [12661.019614] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12661.309879] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [12663.039299] Key type lgssc registered [12664.387791] Lustre: Echo OBD driver; http://www.lustre.org/ [12673.027303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12673.039213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12674.653522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12679.912176] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12684.769763] LustreError: 428232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12684.781359] LustreError: 428232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12689.012937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12689.417156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12694.535016] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12702.827066] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12708.439975] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12716.182516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12716.553594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12719.620634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12722.608708] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12729.068749] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12734.946677] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12734.964764] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12736.783951] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [12736.794984] Lustre: Skipped 1 previous similar message [12739.831869] Lustre: server umount lustre-MDT0001 complete [12740.065499] LustreError: 428228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12740.081600] LustreError: 428228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12745.187586] LustreError: 428233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12745.203075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12745.208353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12745.226101] Lustre: Skipped 1 previous similar message [12745.230602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12750.312438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12750.315861] LustreError: 428971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12751.056339] Lustre: server umount lustre-OST0000 complete [12754.834505] Lustre: server umount lustre-MDT0000 complete [12768.665984] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [12771.576710] Key type lgssc unregistered [12771.873622] LNet: 431918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12771.877764] LNetError: 431918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12772.902382] LNet: Removed LNI 192.168.201.136@tcp [12773.948312] Key type .llcrypt unregistered [12773.950299] Key type ._llcrypt unregistered [12798.841511] Key type ._llcrypt registered [12798.870241] Key type .llcrypt registered [12799.080796] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [12814.555674] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12815.755684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12815.976210] alg: No test for adler32 (adler32-zlib) [12817.063412] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12817.372676] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [12819.114485] Key type lgssc registered [12820.212351] Lustre: Echo OBD driver; http://www.lustre.org/ [12862.397686] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [12879.704740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12879.724830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12880.917772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12880.938604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12881.043628] Lustre: lustre-MDT0000: new disk, initializing [12881.119536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12881.139465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12885.063632] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12899.243993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12899.481331] Lustre: lustre-OST0000: new disk, initializing [12899.485148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12899.488196] Lustre: Skipped 1 previous similar message [12899.491598] Lustre: 436957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12899.556658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12904.970977] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12908.633069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12908.639498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12908.666838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12912.707391] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12927.008196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12927.130134] Lustre: 435938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12927.201657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12927.309822] Lustre: lustre-MDT0001: new disk, initializing [12927.525333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12927.590073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12927.608376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12932.417962] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [12942.410610] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12942.609672] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12946.968414] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12947.309767] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12953.057458] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12953.069352] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12953.093706] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12953.106144] Lustre: Skipped 1 previous similar message [12955.435524] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [12958.194457] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12960.527264] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [12960.544795] Lustre: Skipped 1 previous similar message [12965.658464] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.36@tcp (stopping) [12965.681215] Lustre: Skipped 2 previous similar messages [12967.391134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12967.592721] Lustre: server umount lustre-MDT0001 complete [12973.544549] LustreError: 436965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12973.555535] LustreError: 436965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12978.661824] LustreError: 435951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12978.669188] LustreError: 435951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12983.783635] LustreError: 436965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12983.814337] LustreError: 436965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12988.897478] LustreError: 435952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12988.900978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12988.953132] Lustre: Skipped 1 previous similar message [12988.963453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12994.037138] LustreError: 436944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12994.389895] Lustre: server umount lustre-OST0000 complete [13006.164930] Lustre: server umount lustre-MDT0000 complete [13023.202628] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13027.488711] Key type lgssc unregistered [13027.857361] LNet: 440703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13027.871890] LNetError: 440703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13027.906398] LNet: Removed LNI 192.168.201.136@tcp [13028.585563] Key type .llcrypt unregistered [13028.588778] Key type ._llcrypt unregistered [13066.408907] Key type ._llcrypt registered [13066.410555] Key type .llcrypt registered [13066.597607] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [13066.941432] Key type .llcrypt unregistered [13066.942778] Key type ._llcrypt unregistered [13087.391272] Key type ._llcrypt registered [13087.393655] Key type .llcrypt registered [13087.547877] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [13088.740471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13088.802637] alg: No test for adler32 (adler32-zlib) [13089.995396] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13090.365401] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [13092.159389] Key type lgssc registered [13093.211691] Lustre: Echo OBD driver; http://www.lustre.org/ [13123.039254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13123.062199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13124.353297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13124.401914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13124.464581] Lustre: lustre-MDT0000: new disk, initializing [13124.538813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13124.557290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13129.111247] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13140.167986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13140.289492] Lustre: 444075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13140.362063] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13140.366239] Lustre: Skipped 1 previous similar message [13140.470574] Lustre: lustre-MDT0001: new disk, initializing [13140.630893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13140.682967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13140.689057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13146.025751] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13154.712809] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13160.042641] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13167.884407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13168.175637] Lustre: lustre-OST0000: new disk, initializing [13168.181382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13168.192624] Lustre: 446044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13168.360900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13176.092465] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13177.436560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13177.446606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13177.560082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13186.175570] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13193.188103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13193.204959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13193.215294] Lustre: Skipped 1 previous similar message [13193.232986] Lustre: Skipped 2 previous similar messages [13198.305992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13198.321215] Lustre: Skipped 2 previous similar messages [13198.531264] Lustre: server umount lustre-MDT0000 complete [13202.507165] LustreError: 444065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449928 with bad export cookie 7417512593620189191 [13202.508534] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13202.517820] LustreError: 444065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13202.867448] Lustre: server umount lustre-MDT0001 complete [13217.337183] Lustre: server umount lustre-OST0000 complete [13228.497494] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:39:13 (1788449953) [13236.100581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13236.657255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13241.631307] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13241.836175] LustreError: 448070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13241.858703] LustreError: 448070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13245.925231] LustreError: 448071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13250.651548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13251.052299] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13251.238971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13256.055421] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13263.881827] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13269.791366] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13290.482426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13290.867218] Lustre: Found index 0 for lustre-OST0000, updating log [13291.168866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13297.918828] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13306.463694] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13307.628276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13318.117149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13318.121187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13318.127549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13318.145430] Lustre: Skipped 2 previous similar messages [13324.512941] Lustre: server umount lustre-OST0000 complete [13343.203104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13343.210800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13343.230197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13343.238807] Lustre: Skipped 3 previous similar messages [13345.321177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13347.045921] Lustre: server umount lustre-MDT0000 complete [13350.832171] LustreError: 448050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450076 with bad export cookie 7417512593620190584 [13350.844581] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13351.235055] Lustre: server umount lustre-MDT0001 complete [13354.656476] LNet: 1 local NIs in recovery (showing 1): 192.168.201.136@tcp [13361.320545] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:41:25 (1788450085) [13982.623388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13982.956557] LustreError: 454481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13983.001035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13987.449790] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [13988.330517] LustreError: 454482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13993.442869] LustreError: 454481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13995.165126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13995.489474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13999.021761] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14006.471451] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14012.764276] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14031.663799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14032.004790] Lustre: Found index 0 for lustre-OST0000, updating log [14032.408931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14037.488687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [14038.452471] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14045.777371] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14045.860921] LNet: 1 local NIs in recovery (showing 1): 192.168.201.136@tcp [14049.130662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14049.154210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14049.156884] Lustre: Skipped 4 previous similar messages [14054.367392] Lustre: server umount lustre-OST0000 complete [14058.977452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14058.993446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14058.997719] Lustre: Skipped 1 previous similar message [14061.026861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14061.041859] Lustre: Skipped 3 previous similar messages [14065.122043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14065.134469] Lustre: Skipped 7 previous similar messages [14072.288914] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14072.734227] Lustre: server umount lustre-MDT0000 complete [14073.323811] LustreError: 455209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14073.378109] LustreError: 455209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14077.415220] LustreError: 454481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14077.439145] LustreError: 454481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [14077.702842] LustreError: 454462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450803 with bad export cookie 7417512593620191669 [14077.729679] LustreError: 454462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [14077.735600] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14078.514865] Lustre: server umount lustre-MDT0001 complete [14109.510818] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [14120.118482] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [14156.212444] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 11:54:40 (1788450880) [14176.554480] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [14179.481361] Key type lgssc unregistered [14179.784724] LNet: 461090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14179.794069] LNetError: 461090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14292.455466] LNet: Removed LNI 192.168.201.136@tcp [14293.317728] Key type .llcrypt unregistered [14293.319922] Key type ._llcrypt unregistered [14308.618954] Key type ._llcrypt registered [14308.622185] Key type .llcrypt registered [14308.701892] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [14309.599736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14309.761107] alg: No test for adler32 (adler32-zlib) [14310.799249] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14311.034928] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [14312.727387] Key type lgssc registered [14313.766213] Lustre: Echo OBD driver; http://www.lustre.org/ [14317.440838] LNet: Added LNI 192.168.201.136@tcp100 [8/256/0/180] [14318.168969] LNet: Added LNI 192.168.201.136@tcp101 [8/256/0/180] [14318.967140] LNet: Added LNI 192.168.201.136@tcp102 [8/256/0/180] [14319.842422] LNet: Added LNI 192.168.201.136@tcp103 [8/256/0/180] [14320.692205] LNet: Added LNI 192.168.201.136@tcp104 [8/256/0/180] [14321.533764] LNet: Added LNI 192.168.201.136@tcp105 [8/256/0/180] [14322.519069] LNet: Added LNI 192.168.201.136@tcp106 [8/256/0/180] [14323.327386] LNet: Added LNI 192.168.201.136@tcp107 [8/256/0/180] [14324.057618] LNet: Added LNI 192.168.201.136@tcp108 [8/256/0/180] [14324.860211] LNet: Added LNI 192.168.201.136@tcp109 [8/256/0/180] [14325.565806] LNet: Added LNI 192.168.201.136@tcp110 [8/256/0/180] [14326.279632] LNet: Added LNI 192.168.201.136@tcp111 [8/256/0/180] [14327.015952] LNet: Added LNI 192.168.201.136@tcp112 [8/256/0/180] [14327.796331] LNet: Added LNI 192.168.201.136@tcp113 [8/256/0/180] [14328.645304] LNet: Added LNI 192.168.201.136@tcp114 [8/256/0/180] [14329.539810] LNet: Added LNI 192.168.201.136@tcp115 [8/256/0/180] [14330.433571] LNet: Added LNI 192.168.201.136@tcp116 [8/256/0/180] [14331.199367] LNet: Added LNI 192.168.201.136@tcp117 [8/256/0/180] [14331.910909] LNet: Added LNI 192.168.201.136@tcp118 [8/256/0/180] [14332.930473] LNet: Added LNI 192.168.201.136@tcp119 [8/256/0/180] [14333.753532] LNet: Added LNI 192.168.201.136@tcp120 [8/256/0/180] [14334.756793] LNet: Added LNI 192.168.201.136@tcp121 [8/256/0/180] [14335.577858] LNet: Added LNI 192.168.201.136@tcp122 [8/256/0/180] [14336.431664] LNet: Added LNI 192.168.201.136@tcp123 [8/256/0/180] [14337.197823] LNet: Added LNI 192.168.201.136@tcp124 [8/256/0/180] [14337.996790] LNet: Added LNI 192.168.201.136@tcp125 [8/256/0/180] [14338.724684] LNet: Added LNI 192.168.201.136@tcp126 [8/256/0/180] [14339.584405] LNet: Added LNI 192.168.201.136@tcp127 [8/256/0/180] [14340.326693] LNet: Added LNI 192.168.201.136@tcp128 [8/256/0/180] [14341.296257] LNet: Added LNI 192.168.201.136@tcp129 [8/256/0/180] [14341.990223] LNet: Added LNI 192.168.201.136@tcp130 [8/256/0/180] [14342.834024] LNet: Added LNI 192.168.201.136@tcp131 [8/256/0/180] [14343.760374] LNet: Added LNI 192.168.201.136@tcp132 [8/256/0/180] [14344.602409] LNet: Added LNI 192.168.201.136@tcp133 [8/256/0/180] [14345.284793] LNet: Added LNI 192.168.201.136@tcp134 [8/256/0/180] [14346.118190] LNet: Added LNI 192.168.201.136@tcp135 [8/256/0/180] [14354.604711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14354.656194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14355.896818] Lustre: 464119:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [14355.995756] LNetError: 464142:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [14356.008918] LNetError: Protocol error connecting to 192.168.201.136@tcp118 at host 192.168.201.136:988: is it running a compatible version of Lustre? [14356.032017] LNetError: Skipped 11 previous similar messages [14356.096217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14356.120634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14356.127670] LNet: 10 local NIs in recovery (showing 10): 192.168.201.136@tcp122, 192.168.201.136@tcp123, 192.168.201.136@tcp124, 192.168.201.136@tcp125, 192.168.201.136@tcp126, 192.168.201.136@tcp127, 192.168.201.136@tcp119, 192.168.201.136@tcp129, 192.168.201.136@tcp128, 192.168.201.136@tcp118 [14356.196452] Lustre: lustre-MDT0000: new disk, initializing [14356.251724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14356.263877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14357.223526] LNetError: 461580:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [14357.253327] LNetError: 461580:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14357.267966] LNetError: Protocol error connecting to 192.168.201.136@tcp122 at host 192.168.201.136:988: is it running a compatible version of Lustre? [14357.294171] LNetError: Skipped 3 previous similar messages [14359.267985] LNetError: 464150:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [14359.304987] LNetError: 464150:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14359.327216] LNetError: Protocol error connecting to 192.168.201.136@tcp122 at host 192.168.201.136:988: is it running a compatible version of Lustre? [14360.941873] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14363.371643] LNetError: 464142:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [14363.377287] LNetError: Protocol error connecting to 192.168.201.136@tcp122 at host 192.168.201.136:988: is it running a compatible version of Lustre? [14371.561768] LNetError: 464233:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [14371.573871] LNetError: Protocol error connecting to 192.168.201.136@tcp122 at host 192.168.201.136:988: is it running a compatible version of Lustre? [14372.250865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14372.434771] Lustre: 464135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14372.494566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14372.501581] Lustre: Skipped 1 previous similar message [14372.644857] Lustre: lustre-MDT0001: new disk, initializing [14372.698267] Lustre: 464935:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [14372.706148] Lustre: 464935:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14372.919758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14372.950170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14372.967569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14374.269435] Lustre: 464987:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [14374.298809] Lustre: 464987:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14378.243345] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14386.021912] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14387.891847] LNetError: 461577:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [14387.898352] LNetError: Protocol error connecting to 192.168.201.136@tcp123 at host 192.168.201.136:988: is it running a compatible version of Lustre? [14391.838671] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14398.514673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14398.866675] Lustre: lustre-OST0000: new disk, initializing [14398.872039] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14398.880928] Lustre: 466108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14398.974720] Lustre: 466113:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [14398.983635] Lustre: 466113:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14399.032086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14405.681612] Lustre: 466422:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.136@tcp: can't add NID 192.168.201.136@tcp130: rc = -75 [14405.687898] Lustre: 466422:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14405.769221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14405.784110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14405.958905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14406.180545] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14410.407430] LNetError: 464233:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [14410.424297] LNetError: 464233:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14410.430529] LNetError: Protocol error connecting to 192.168.201.136@tcp120 at host 192.168.201.136:988: is it running a compatible version of Lustre? [14410.439797] LNetError: Skipped 7 previous similar messages [14416.136744] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14416.607383] LNet: 1 local NIs in recovery (showing 1): 192.168.201.136@tcp122 [14426.594234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14426.595613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14426.612153] LustreError: Skipped 1 previous similar message [14426.626902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14428.907943] Lustre: server umount lustre-OST0000 complete [14434.272619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14434.276669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14434.303650] Lustre: Skipped 1 previous similar message [14434.324200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14434.328837] Lustre: Skipped 1 previous similar message [14436.833824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14436.835657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14436.857472] Lustre: Skipped 1 previous similar message [14440.047605] Lustre: server umount lustre-MDT0000 complete [14444.397602] LustreError: 464944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451170 with bad export cookie 13453686800738861264 [14444.407396] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14444.412536] LustreError: 464944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14444.888839] Lustre: server umount lustre-MDT0001 complete [14451.492419] LNetError: 461577:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.136 [14451.511271] LNetError: 461577:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14451.523803] LNetError: Protocol error connecting to 192.168.201.136@tcp123 at host 192.168.201.136:988: is it running a compatible version of Lustre? [14451.540803] LNetError: Skipped 9 previous similar messages [14456.529429] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [14459.666630] Key type lgssc unregistered [14460.094374] LNet: 468018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14460.106572] LNetError: 468018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14460.116424] LNet: Removed LNI 192.168.201.136@tcp [14461.151302] LNet: Removed LNI 192.168.201.136@tcp100 [14461.153272] LNetError: 468018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14461.167946] LNetError: 468018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14461.181098] LNet: Removed LNI 192.168.201.136@tcp101 [14462.240164] LNet: Removed LNI 192.168.201.136@tcp102 [14462.244499] LNetError: 468018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14462.254781] LNetError: 468018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14462.270976] LNet: Removed LNI 192.168.201.136@tcp103 [14463.329181] LNet: Removed LNI 192.168.201.136@tcp104 [14464.352248] LNet: Removed LNI 192.168.201.136@tcp105 [14464.366582] LNetError: 468018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14464.377870] LNetError: 468018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14464.395339] LNet: Removed LNI 192.168.201.136@tcp106 [14465.439801] LNet: Removed LNI 192.168.201.136@tcp107 [14466.463733] LNet: Removed LNI 192.168.201.136@tcp108 [14467.491942] LNet: Removed LNI 192.168.201.136@tcp109 [14468.511174] LNet: Removed LNI 192.168.201.136@tcp110 [14468.517798] LNetError: 468018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14468.525031] LNetError: 468018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14469.535366] LNet: Removed LNI 192.168.201.136@tcp111 [14470.560397] LNet: Removed LNI 192.168.201.136@tcp112 [14471.583276] LNet: Removed LNI 192.168.201.136@tcp113 [14472.607167] LNet: Removed LNI 192.168.201.136@tcp114 [14473.631145] LNet: Removed LNI 192.168.201.136@tcp115 [14474.655947] LNet: Removed LNI 192.168.201.136@tcp116 [14475.679369] LNet: Removed LNI 192.168.201.136@tcp117 [14476.703137] LNet: Removed LNI 192.168.201.136@tcp118 [14476.704564] LNetError: 468018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14476.706617] LNetError: 468018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [14476.708895] LNet: Removed LNI 192.168.201.136@tcp119 [14477.727212] LNet: Removed LNI 192.168.201.136@tcp120 [14478.752559] LNet: Removed LNI 192.168.201.136@tcp121 [14478.754878] LNet: Removed LNI 192.168.201.136@tcp122 [14479.775396] LNet: Removed LNI 192.168.201.136@tcp123 [14480.799104] LNet: Removed LNI 192.168.201.136@tcp124 [14481.825684] LNet: Removed LNI 192.168.201.136@tcp125 [14482.848130] LNet: Removed LNI 192.168.201.136@tcp126 [14483.935324] LNet: Removed LNI 192.168.201.136@tcp127 [14484.959174] LNet: Removed LNI 192.168.201.136@tcp128 [14485.983128] LNet: Removed LNI 192.168.201.136@tcp129 [14485.996607] LNet: Removed LNI 192.168.201.136@tcp130 [14486.000048] LNet: Removed LNI 192.168.201.136@tcp131 [14486.002931] LNet: Removed LNI 192.168.201.136@tcp132 [14486.025384] LNet: Removed LNI 192.168.201.136@tcp133 [14486.032853] LNet: Removed LNI 192.168.201.136@tcp134 [14486.067519] LNet: Removed LNI 192.168.201.136@tcp135 [14486.711294] Key type .llcrypt unregistered [14486.714709] Key type ._llcrypt unregistered [14507.350980] Key type ._llcrypt registered [14507.352870] Key type .llcrypt registered [14507.435781] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [14521.776338] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [14523.105541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14523.126679] alg: No test for adler32 (adler32-zlib) [14524.151320] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14524.391392] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [14526.047187] Key type lgssc registered [14527.323182] Lustre: Echo OBD driver; http://www.lustre.org/ [14567.273505] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:01:31 (1788451291) [14586.642491] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [14590.042791] Key type lgssc unregistered [14590.330725] LNet: 472130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14590.343676] LNetError: 472130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14590.369282] LNet: Removed LNI 192.168.201.136@tcp [14591.372779] Key type .llcrypt unregistered [14591.374858] Key type ._llcrypt unregistered [14608.952271] Key type ._llcrypt registered [14608.953905] Key type .llcrypt registered [14609.094879] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [14610.667331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14610.698243] alg: No test for adler32 (adler32-zlib) [14611.757803] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14611.933608] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [14613.599313] Key type lgssc registered [14614.854811] Lustre: Echo OBD driver; http://www.lustre.org/ [14625.134352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14625.175364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14626.490214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14626.544573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14626.605512] Lustre: lustre-MDT0000: new disk, initializing [14626.739222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14626.763592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14632.155851] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14642.568983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14642.688409] Lustre: 473341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14642.736406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14642.739726] Lustre: Skipped 1 previous similar message [14642.806747] Lustre: lustre-MDT0001: new disk, initializing [14642.958860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14643.062172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14643.074911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14648.265045] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14656.721415] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14662.658323] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14669.573456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14669.898207] Lustre: lustre-OST0000: new disk, initializing [14669.908157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14669.925158] Lustre: 475307:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14670.033866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14675.987777] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [14679.117637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14679.124590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14679.210291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14684.645953] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14688.284139] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing lnet_if_list [14693.637804] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14693.667890] LNet: Added LNI 192.168.201.136@tcp10 [8/256/0/180] [14699.993655] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14700.037110] LNet: Added LNI 192.168.201.136@tcp11 [8/256/0/180] [14704.714120] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14704.737655] LNet: Added LNI 192.168.201.136@tcp12 [8/256/0/180] [14708.692569] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14708.725926] LNet: Added LNI 192.168.201.136@tcp13 [8/256/0/180] [14713.298808] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14713.338106] LNet: Added LNI 192.168.201.136@tcp14 [8/256/0/180] [14717.816352] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14717.843815] LNet: Added LNI 192.168.201.136@tcp15 [8/256/0/180] [14722.715545] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14722.758921] LNet: Added LNI 192.168.201.136@tcp16 [8/256/0/180] [14727.791758] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14727.814133] LNet: Added LNI 192.168.201.136@tcp17 [8/256/0/180] [14733.102848] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14733.122578] LNet: Added LNI 192.168.201.136@tcp18 [8/256/0/180] [14738.494255] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14738.551442] LNet: Added LNI 192.168.201.136@tcp19 [8/256/0/180] [14743.608921] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14743.637096] LNet: Added LNI 192.168.201.136@tcp20 [8/256/0/180] [14748.925570] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14748.947702] LNet: Added LNI 192.168.201.136@tcp21 [8/256/0/180] [14754.553991] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14754.596599] LNet: Added LNI 192.168.201.136@tcp22 [8/256/0/180] [14760.933810] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14760.974207] LNet: Added LNI 192.168.201.136@tcp23 [8/256/0/180] [14767.285704] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14767.325592] LNet: Added LNI 192.168.201.136@tcp24 [8/256/0/180] [14775.192345] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14775.233621] LNet: Added LNI 192.168.201.136@tcp25 [8/256/0/180] [14782.179874] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14782.203714] LNet: Added LNI 192.168.201.136@tcp26 [8/256/0/180] [14789.309124] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14789.331032] LNet: Added LNI 192.168.201.136@tcp27 [8/256/0/180] [14795.837737] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14795.872970] LNet: Added LNI 192.168.201.136@tcp28 [8/256/0/180] [14802.633347] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14802.654458] LNet: Added LNI 192.168.201.136@tcp29 [8/256/0/180] [14808.588280] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14808.634978] LNet: Added LNI 192.168.201.136@tcp30 [8/256/0/180] [14815.988725] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14816.028511] LNet: Added LNI 192.168.201.136@tcp31 [8/256/0/180] [14824.417200] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14824.462758] LNet: Added LNI 192.168.201.136@tcp32 [8/256/0/180] [14831.990029] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14832.013506] LNet: Added LNI 192.168.201.136@tcp33 [8/256/0/180] [14839.366432] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14839.408472] LNet: Added LNI 192.168.201.136@tcp34 [8/256/0/180] [14846.772605] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14846.802942] LNet: Added LNI 192.168.201.136@tcp35 [8/256/0/180] [14853.932888] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14853.977514] LNet: Added LNI 192.168.201.136@tcp36 [8/256/0/180] [14861.495631] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14861.571835] LNet: Added LNI 192.168.201.136@tcp37 [8/256/0/180] [14868.056651] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14868.095697] LNet: Added LNI 192.168.201.136@tcp38 [8/256/0/180] [14875.800631] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14875.860138] LNet: Added LNI 192.168.201.136@tcp39 [8/256/0/180] [14884.358518] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14884.371864] LNet: Added LNI 192.168.201.136@tcp40 [8/256/0/180] [14893.124881] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14893.149465] LNet: Added LNI 192.168.201.136@tcp41 [8/256/0/180] [14901.419522] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14901.451740] LNet: Added LNI 192.168.201.136@tcp42 [8/256/0/180] [14907.202801] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14907.239436] LNet: Added LNI 192.168.201.136@tcp43 [8/256/0/180] [14915.586363] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14915.618137] LNet: Added LNI 192.168.201.136@tcp44 [8/256/0/180] [14926.618260] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14926.643750] LNet: Added LNI 192.168.201.136@tcp45 [8/256/0/180] [14935.708630] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14935.777104] LNet: Added LNI 192.168.201.136@tcp46 [8/256/0/180] [14945.180532] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14945.211418] LNet: Added LNI 192.168.201.136@tcp47 [8/256/0/180] [14954.929572] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14954.962496] LNet: Added LNI 192.168.201.136@tcp48 [8/256/0/180] [14965.275682] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14965.301567] LNet: Added LNI 192.168.201.136@tcp49 [8/256/0/180] [14975.375113] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14975.430918] LNet: Added LNI 192.168.201.136@tcp50 [8/256/0/180] [15001.571826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15001.577967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15001.601665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15002.081790] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15002.105323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15002.114286] Lustre: Skipped 2 previous similar messages [15004.344748] Lustre: server umount lustre-MDT0000 complete [15008.793944] LustreError: 473330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451734 with bad export cookie 6171894703154422224 [15008.801182] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15009.322195] Lustre: server umount lustre-MDT0001 complete [15028.703912] Lustre: 472614:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451738/real 1788451738] req@00000000cf3e2b3d x1875327150370176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788451754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15028.744670] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15028.761222] Lustre: Skipped 1 previous similar message [15029.731180] Lustre: 472613:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451739/real 1788451739] req@000000007176ab96 x1875327150370304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788451755 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15029.863776] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [15032.875640] Lustre: 472611:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451743/real 1788451743] req@000000003802593c x1875327150370432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788451759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15038.943286] Lustre: 472614:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451749/real 1788451749] req@00000000d61d790d x1875327150370816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788451765 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15038.975976] Lustre: 472614:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15040.860819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15044.063900] Lustre: 472613:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451754/real 1788451754] req@00000000ba28f926 x1875327150370944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788451770 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15053.618087] LustreError: 475301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15053.629482] LustreError: 475301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15053.663400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15058.839953] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15058.853034] LustreError: 475302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15058.854824] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15064.033339] LustreError: 475301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15067.789068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15068.138149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15073.270806] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15073.427037] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15081.452679] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15087.655887] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15109.091092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15109.091692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15109.095194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15109.134775] Lustre: Skipped 1 previous similar message [15114.214362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15114.220218] Lustre: Skipped 2 previous similar messages [15115.019182] Lustre: server umount lustre-OST0000 complete [15119.332525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15119.345569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15119.349021] Lustre: Skipped 1 previous similar message [15119.361122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15125.725853] Lustre: server umount lustre-MDT0000 complete [15129.580460] LustreError: 499567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15129.599157] LustreError: 499567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15129.707153] LustreError: 473331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451855 with bad export cookie 6171894703154424863 [15129.713646] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15129.718921] LustreError: 473331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15130.070859] Lustre: server umount lustre-MDT0001 complete [15142.135814] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [15146.085722] Key type lgssc unregistered [15146.476438] LNet: 502365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15146.485986] LNetError: 502365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15146.492511] LNet: Removed LNI 192.168.201.136@tcp [15146.495516] LNet: Removed LNI 192.168.201.136@tcp10 [15146.498718] LNet: Removed LNI 192.168.201.136@tcp11 [15146.501858] LNet: Removed LNI 192.168.201.136@tcp12 [15146.504775] LNet: Removed LNI 192.168.201.136@tcp13 [15146.507591] LNet: Removed LNI 192.168.201.136@tcp14 [15146.511526] LNet: Removed LNI 192.168.201.136@tcp15 [15146.513557] LNet: Removed LNI 192.168.201.136@tcp16 [15146.516480] LNet: Removed LNI 192.168.201.136@tcp17 [15146.519828] LNet: Removed LNI 192.168.201.136@tcp18 [15146.523085] LNet: Removed LNI 192.168.201.136@tcp19 [15146.524869] LNet: Removed LNI 192.168.201.136@tcp20 [15146.527664] LNet: Removed LNI 192.168.201.136@tcp21 [15146.531220] LNet: Removed LNI 192.168.201.136@tcp22 [15146.533479] LNet: Removed LNI 192.168.201.136@tcp23 [15146.536599] LNet: Removed LNI 192.168.201.136@tcp24 [15146.539251] LNet: Removed LNI 192.168.201.136@tcp25 [15146.544256] LNet: Removed LNI 192.168.201.136@tcp26 [15146.547180] LNet: Removed LNI 192.168.201.136@tcp27 [15146.549867] LNet: Removed LNI 192.168.201.136@tcp28 [15146.552618] LNet: Removed LNI 192.168.201.136@tcp29 [15146.555181] LNet: Removed LNI 192.168.201.136@tcp30 [15146.559188] LNet: Removed LNI 192.168.201.136@tcp31 [15146.563292] LNet: Removed LNI 192.168.201.136@tcp32 [15146.566519] LNet: Removed LNI 192.168.201.136@tcp33 [15146.569588] LNet: Removed LNI 192.168.201.136@tcp34 [15146.573503] LNet: Removed LNI 192.168.201.136@tcp35 [15146.576622] LNet: Removed LNI 192.168.201.136@tcp36 [15146.579491] LNet: Removed LNI 192.168.201.136@tcp37 [15146.586166] LNet: Removed LNI 192.168.201.136@tcp38 [15146.590133] LNet: Removed LNI 192.168.201.136@tcp39 [15146.595702] LNet: Removed LNI 192.168.201.136@tcp40 [15146.599310] LNet: Removed LNI 192.168.201.136@tcp41 [15146.602711] LNet: Removed LNI 192.168.201.136@tcp42 [15146.605440] LNet: Removed LNI 192.168.201.136@tcp43 [15146.607737] LNet: Removed LNI 192.168.201.136@tcp44 [15146.610687] LNet: Removed LNI 192.168.201.136@tcp45 [15146.613708] LNet: Removed LNI 192.168.201.136@tcp46 [15146.616765] LNet: Removed LNI 192.168.201.136@tcp47 [15146.620351] LNet: Removed LNI 192.168.201.136@tcp48 [15146.623477] LNet: Removed LNI 192.168.201.136@tcp49 [15146.640300] LNet: Removed LNI 192.168.201.136@tcp50 [15147.433221] Key type .llcrypt unregistered [15147.437425] Key type ._llcrypt unregistered [15171.212750] Key type ._llcrypt registered [15171.215976] Key type .llcrypt registered [15171.386088] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [15187.821898] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [15188.853955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15188.898218] alg: No test for adler32 (adler32-zlib) [15190.220690] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15190.532248] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [15192.175403] Key type lgssc registered [15193.323522] Lustre: Echo OBD driver; http://www.lustre.org/ [15230.385973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15230.429905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15231.775870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15231.805791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15231.942283] Lustre: lustre-MDT0000: new disk, initializing [15232.016960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15232.034605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15236.894312] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15246.658294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15246.810321] Lustre: 505560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15246.866798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15246.872819] Lustre: Skipped 1 previous similar message [15246.942234] Lustre: lustre-MDT0001: new disk, initializing [15247.045668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15247.091051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15247.107888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15252.047091] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15261.838909] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15267.102292] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15275.187267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15275.354568] Lustre: *** cfs_fail_loc=90f, val=0*** [15275.439350] Lustre: lustre-OST0000: new disk, initializing [15275.442897] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15275.451725] Lustre: 507578:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15275.595146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15282.348493] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15284.825048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15284.842568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15284.980293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15292.419640] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15305.712806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15305.719376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15305.726826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15305.738415] Lustre: Skipped 1 previous similar message [15305.744454] LustreError: Skipped 1 previous similar message [15307.183580] Lustre: server umount lustre-OST0000 complete [15310.825599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15310.835810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15310.842633] Lustre: Skipped 2 previous similar messages [15315.937764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15321.059113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15321.071883] Lustre: Skipped 2 previous similar messages [15325.151130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15325.162213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15325.170593] Lustre: Skipped 1 previous similar message [15325.327402] Lustre: server umount lustre-MDT0000 complete [15329.367229] LustreError: 507577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452055 with bad export cookie 15271471254691792638 [15329.384982] LustreError: 507577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15329.386950] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15329.745620] Lustre: server umount lustre-MDT0001 complete [15339.920445] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [15342.957944] Key type lgssc unregistered [15343.166974] LNet: 509486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15343.174198] LNetError: 509486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15343.185150] LNet: Removed LNI 192.168.201.136@tcp [15344.100486] Key type .llcrypt unregistered [15344.103360] Key type ._llcrypt unregistered [15363.580398] Key type ._llcrypt registered [15363.582754] Key type .llcrypt registered [15363.712066] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [15364.668299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15364.720538] alg: No test for adler32 (adler32-zlib) [15366.165572] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15366.469605] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [15368.177066] Key type lgssc registered [15369.468838] Lustre: Echo OBD driver; http://www.lustre.org/ [15377.871568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15377.888309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15379.281709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15384.421961] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15389.666108] LustreError: 510750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15389.675950] LustreError: 510750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15393.032732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15393.373039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15397.915951] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15405.583694] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15410.972456] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15417.713945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15418.051596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15424.138314] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15432.338282] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15443.507947] loop0: detected capacity change from 0 to 204800000 [15443.730373] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15443.745921] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15443.758409] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15449.898891] loop0: detected capacity change from 0 to 204800000 [15450.111466] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15450.132695] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15450.148154] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15455.449482] loop0: detected capacity change from 0 to 204800000 [15460.995762] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15461.270580] Lustre: fs15246-OST0000: new disk, initializing [15461.279796] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15461.298934] Lustre: 514344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15461.363863] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15467.298700] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15475.895503] loop1: detected capacity change from 0 to 204800000 [15483.310860] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15483.484541] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15483.562487] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15483.698359] Lustre: fs15246-MDT0000: new disk, initializing [15483.963695] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15484.012865] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15484.234812] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15485.288673] Lustre: 515464:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000015b1d748 x1875327940866432/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788452227 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15489.721381] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15490.419171] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15490.436574] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15490.489913] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15515.621952] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15515.641476] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15518.014748] Lustre: server umount fs15246-MDT0000 complete [15520.758825] LustreError: 510745:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15525.868676] LustreError: 516276:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15526.985128] Lustre: server umount fs15246-OST0000 complete [15539.175233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15539.206818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15539.215316] Lustre: Skipped 1 previous similar message [15544.292597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15544.297691] Lustre: Skipped 1 previous similar message [15545.355117] Lustre: server umount lustre-OST0000 complete [15551.969420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15551.979122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15551.987190] Lustre: Skipped 1 previous similar message [15551.992928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15554.544191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15559.648778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15559.658623] Lustre: Skipped 3 previous similar messages [15564.255155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15564.432288] Lustre: server umount lustre-MDT0000 complete [15564.773978] LustreError: 511757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15564.790892] LustreError: 511757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15568.607130] LustreError: 510730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452294 with bad export cookie 15023606664185994768 [15568.611685] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15568.625562] LustreError: 510730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15568.973461] Lustre: server umount lustre-MDT0001 complete [15581.578389] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [15584.368523] Key type lgssc unregistered [15585.011943] LNet: 518562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15585.021082] LNetError: 518562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15585.052417] LNet: Removed LNI 192.168.201.136@tcp [15585.949766] Key type .llcrypt unregistered [15585.954323] Key type ._llcrypt unregistered [15605.022510] Key type ._llcrypt registered [15605.025225] Key type .llcrypt registered [15605.134768] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [15606.688462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15606.766370] alg: No test for adler32 (adler32-zlib) [15607.874690] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15608.108188] LNet: Added LNI 192.168.201.136@tcp [8/256/0/180] [15609.752595] Key type lgssc registered [15611.259157] Lustre: Echo OBD driver; http://www.lustre.org/ [15652.580576] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [15662.073437] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [15695.468740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15695.504250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15696.792184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15696.857850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15696.959321] Lustre: lustre-MDT0000: new disk, initializing [15697.168675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15697.195687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15701.344631] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15710.616720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15710.728612] Lustre: 523028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15710.775326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15710.778307] Lustre: Skipped 1 previous similar message [15710.835905] Lustre: lustre-MDT0001: new disk, initializing [15711.012206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15711.058638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15711.072700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15714.949184] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15723.948823] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15730.876595] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15738.415215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15738.634630] Lustre: lustre-OST0000: new disk, initializing [15738.638369] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15738.643626] Lustre: 524994:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15738.830561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15745.626766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15745.633652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15745.806332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15746.140581] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_default_debug -1 all [15755.007584] Lustre: DEBUG MARKER: oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15759.323247] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15759.510790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15764.410931] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15764.660984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15771.617413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15771.618966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15771.630282] LustreError: Skipped 1 previous similar message [15771.636967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15771.652523] Lustre: Skipped 1 previous similar message [15771.675550] Lustre: Skipped 1 previous similar message [15776.744901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15776.756434] Lustre: Skipped 1 previous similar message [15780.831408] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15781.149909] Lustre: server umount lustre-OST0000 complete [15786.976512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15786.980119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15786.994285] Lustre: Skipped 1 previous similar message [15787.005300] Lustre: Skipped 1 previous similar message [15792.048919] Lustre: server umount lustre-MDT0000 complete [15796.088675] LustreError: 523822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452522 with bad export cookie 17269453302835351996 [15796.094720] LustreError: MGC192.168.201.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15796.097880] LustreError: 523822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15796.546254] Lustre: server umount lustre-MDT0001 complete [15822.043349] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing set_hostid [15834.584510] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing load_modules_local [15866.682949] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15621 sec ========= 12:23:11 (1788452591) [15868.386652] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:23:12 (1788452592) === [15871.730980] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:23:16 (1788452596) === [15913.129301] Lustre: DEBUG MARKER: oleg136-server.virtnet: executing unload_modules_local [15916.432385] Key type lgssc unregistered [15916.711655] LNet: 531638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15916.723283] LNetError: 531638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15916.746062] LNet: Removed LNI 192.168.201.136@tcp [15917.796289] Key type .llcrypt unregistered [15917.799077] Key type ._llcrypt unregistered