[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495314670 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003109] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008493] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011134] LSM: Security Framework initializing [ 0.013018] Yama: becoming mindful. [ 0.014039] SELinux: Initializing. [ 0.015061] *** VALIDATE selinux *** [ 0.023197] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027826] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030121] *** VALIDATE tmpfs *** [ 0.031459] *** VALIDATE proc *** [ 0.033149] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035292] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042161] debug: unmapping init [mem 0xffffffffafa59000-0xffffffffafa60fff] [ 0.044177] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045684] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053303] rcu: Hierarchical SRCU implementation. [ 0.055449] smp: Bringing up secondary CPUs ... [ 0.056537] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060757] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.127517] node 0 deferred pages initialised in 62ms [ 0.130174] devtmpfs: initialized [ 0.131194] x86/mm: Memory block size: 128MB [ 0.132464] gcov: version magic: 0x41383552 [ 0.134150] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.137091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.139310] pinctrl core: initialized pinctrl subsystem [ 0.141151] [ 0.141448] ************************************************************* [ 0.142007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144012] ** ** [ 0.146013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.148012] ** ** [ 0.150011] ** This means that this kernel is built to expose internal ** [ 0.151011] ** IOMMU data structures, which may compromise security on ** [ 0.153012] ** your system. ** [ 0.156013] ** ** [ 0.158009] ** If you see this message and you are not debugging the ** [ 0.161018] ** kernel, report this immediately to your vendor! ** [ 0.163012] ** ** [ 0.166012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168011] ************************************************************* [ 0.171690] NET: Registered protocol family 16 [ 0.173419] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.177056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179084] cpuidle: using governor menu [ 0.180755] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182456] PCI: Using configuration type 1 for base access [ 0.185128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189144] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190057] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192049] cryptd: max_cpu_qlen set to 1000 [ 0.193198] ACPI: Added _OSI(Module Device) [ 0.194055] ACPI: Added _OSI(Processor Device) [ 0.194971] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.201056] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209462] ACPI: Interpreter enabled [ 0.210038] ACPI: PM: (supports S0 S3 S4 S5) [ 0.211012] ACPI: Using IOAPIC for interrupt routing [ 0.213093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216433] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.226755] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.228041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.231021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.239463] acpiphp: Slot [2] registered [ 0.241154] acpiphp: Slot [5] registered [ 0.243242] acpiphp: Slot [6] registered [ 0.244160] acpiphp: Slot [7] registered [ 0.246177] acpiphp: Slot [8] registered [ 0.247152] acpiphp: Slot [9] registered [ 0.249153] acpiphp: Slot [10] registered [ 0.250223] acpiphp: Slot [3] registered [ 0.252117] acpiphp: Slot [4] registered [ 0.253084] acpiphp: Slot [11] registered [ 0.254128] acpiphp: Slot [12] registered [ 0.255082] acpiphp: Slot [13] registered [ 0.257116] acpiphp: Slot [14] registered [ 0.258116] acpiphp: Slot [15] registered [ 0.259129] acpiphp: Slot [16] registered [ 0.261222] acpiphp: Slot [17] registered [ 0.262083] acpiphp: Slot [18] registered [ 0.263102] acpiphp: Slot [19] registered [ 0.264080] acpiphp: Slot [20] registered [ 0.265058] acpiphp: Slot [21] registered [ 0.266070] acpiphp: Slot [22] registered [ 0.267090] acpiphp: Slot [23] registered [ 0.268069] acpiphp: Slot [24] registered [ 0.269029] acpiphp: Slot [25] registered [ 0.270009] acpiphp: Slot [26] registered [ 0.271010] acpiphp: Slot [27] registered [ 0.271922] acpiphp: Slot [28] registered [ 0.273080] acpiphp: Slot [29] registered [ 0.275094] acpiphp: Slot [30] registered [ 0.276102] acpiphp: Slot [31] registered [ 0.277062] PCI host bridge to bus 0000:00 [ 0.278081] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.280017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.282022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.284029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.287030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.289029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.290218] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.293950] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.297270] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.308018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.312379] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.315017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.317019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.319017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.321933] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.325064] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.327043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.329718] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.334015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.345756] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.350952] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.356000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.365015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.375020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.406018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.417559] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.429022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.445039] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.462022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.473308] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.484018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.491018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.507022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.523500] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.536017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.543018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.572015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.585367] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.593017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.600027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.619027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.628013] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.639028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.649017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.663019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.674000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.677445] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.680369] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.683387] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.685234] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.690107] iommu: Default domain type: Passthrough [ 0.691438] SCSI subsystem initialized [ 0.693091] ACPI: bus type USB registered [ 0.694088] usbcore: registered new interface driver usbfs [ 0.696049] usbcore: registered new interface driver hub [ 0.697093] usbcore: registered new device driver usb [ 0.698138] pps_core: LinuxPPS API ver. 1 registered [ 0.699007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.701044] PTP clock support registered [ 0.703216] EDAC MC: Ver: 3.0.0 [ 0.705138] PCI: Using ACPI for IRQ routing [ 0.707934] NetLabel: Initializing [ 0.709012] NetLabel: domain hash size = 128 [ 0.710010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.713091] NetLabel: unlabeled traffic allowed by default [ 0.715132] vgaarb: loaded [ 0.718379] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.719014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.724270] clocksource: Switched to clocksource kvm-clock [ 0.835319] VFS: Disk quotas dquot_6.6.0 [ 0.837162] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.839672] *** VALIDATE ramfs *** [ 0.840973] *** VALIDATE hugetlbfs *** [ 0.842541] pnp: PnP ACPI init [ 0.845151] pnp: PnP ACPI: found 6 devices [ 0.863381] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.866580] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.868943] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.871031] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.872704] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.874743] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.877488] NET: Registered protocol family 2 [ 0.880058] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.885127] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.888628] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.894377] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.898605] TCP: Hash tables configured (established 65536 bind 65536) [ 0.901711] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.905158] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.908265] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.911587] NET: Registered protocol family 1 [ 0.914687] RPC: Registered named UNIX socket transport module. [ 0.916818] RPC: Registered udp transport module. [ 0.918568] RPC: Registered tcp transport module. [ 0.920254] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.922455] NET: Registered protocol family 44 [ 0.923939] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.926374] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.928331] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.930735] PCI: CLS 0 bytes, default 64 [ 0.932253] Unpacking initramfs... [ 2.310445] debug: unmapping init [mem 0xffff89817cc54000-0xffff89817ffbffff] [ 2.313302] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.315222] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.318258] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.870043] Initialise system trusted keyrings [ 2.871982] Key type blacklist registered [ 2.874326] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.884927] zbud: loaded [ 2.888558] *** VALIDATE nfs *** [ 2.889978] *** VALIDATE nfs4 *** [ 2.891591] pstore: using deflate compression [ 2.896359] Platform Keyring initialized [ 3.017606] NET: Registered protocol family 38 [ 3.020178] Key type asymmetric registered [ 3.021825] Asymmetric key parser 'x509' registered [ 3.026161] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.028660] io scheduler mq-deadline registered [ 3.030330] io scheduler kyber registered [ 3.032142] io scheduler bfq registered [ 3.034148] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.037501] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.040392] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.043446] ACPI: Power Button [PWRF] [ 3.049489] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.057197] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.072436] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.083057] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.100452] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.128837] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.156756] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.162135] Non-volatile memory driver v1.3 [ 3.163943] Linux agpgart interface v0.103 [ 3.196665] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.198916] vda: detected capacity change from 0 to 76681216 [ 3.216469] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.219933] vdb: detected capacity change from 0 to 1073741824 [ 3.243612] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.246744] vdc: detected capacity change from 0 to 2621440000 [ 3.263827] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.267085] vdd: detected capacity change from 0 to 2621440000 [ 3.283811] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.287107] vde: detected capacity change from 0 to 4294967296 [ 3.302992] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.306078] vdf: detected capacity change from 0 to 4294967296 [ 3.322732] libphy: Fixed MDIO Bus: probed [ 3.333489] usbcore: registered new interface driver usbserial_generic [ 3.336960] usbserial: USB Serial support registered for generic [ 3.339189] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.343097] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.344652] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.346914] mousedev: PS/2 mouse device common for all mice [ 3.349581] rtc_cmos 00:05: RTC can wake from S4 [ 3.353994] rtc_cmos 00:05: registered as rtc0 [ 3.356436] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.357024] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.358681] intel_pstate: CPU model not supported [ 3.366049] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.367311] hid: raw HID events driver (C) Jiri Kosina [ 3.373359] usbcore: registered new interface driver usbhid [ 3.375700] usbhid: USB HID core driver [ 3.375935] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.377374] drop_monitor: Initializing network drop monitor service [ 3.383619] Initializing XFRM netlink socket [ 3.385895] NET: Registered protocol family 10 [ 3.389578] Segment Routing with IPv6 [ 3.391627] NET: Registered protocol family 17 [ 3.394233] mpls_gso: MPLS GSO support [ 3.400392] RAS: Correctable Errors collector initialized. [ 3.402497] AVX version of gcm_enc/dec engaged. [ 3.404343] AES CTR mode by8 optimization enabled [ 3.490026] sched_clock: Marking stable (3490010341, 0)->(4415949175, -925938834) [ 3.493060] registered taskstats version 1 [ 3.494927] Loading compiled-in X.509 certificates [ 3.496532] zswap: loaded using pool lzo/zbud [ 3.520122] Key type big_key registered [ 3.530522] Key type encrypted registered [ 3.531722] ima: No TPM chip found, activating TPM-bypass! [ 3.533499] ima: Allocated hash algorithm: sha1 [ 3.534667] ima: No architecture policies found [ 3.537097] evm: Initialising EVM extended attributes: [ 3.539313] evm: security.selinux [ 3.540687] evm: security.ima [ 3.542098] evm: security.capability [ 3.543595] evm: HMAC attrs: 0x1 [ 3.546200] rtc_cmos 00:05: setting system clock to 2026-09-05 20:33:56 UTC (1788640436) [ 3.553218] debug: unmapping init [mem 0xffffffffb0a03000-0xffffffffb0bfffff] [ 3.557094] debug: unmapping init [mem 0xffffffffaf782000-0xffffffffafa58fff] [ 3.566112] Write protecting the kernel read-only data: 28672k [ 3.570894] debug: unmapping init [mem 0xffffffffade03000-0xffffffffadffffff] [ 3.574480] debug: unmapping init [mem 0xffffffffae714000-0xffffffffae7fffff] [ 3.606750] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.612780] systemd[1]: Detected virtualization kvm. [ 3.614221] systemd[1]: Detected architecture x86-64. [ 3.615494] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.642558] systemd[1]: No hostname configured. [ 3.643780] systemd[1]: Set hostname to . [ 3.645274] random: systemd: uninitialized urandom read (16 bytes read) [ 3.647202] systemd[1]: Initializing machine ID from random generator. [ 3.789291] random: systemd: uninitialized urandom read (16 bytes read) [ 3.792185] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.796868] random: systemd: uninitialized urandom read (16 bytes read) [ 3.799258] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.804488] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.477293] device-mapper: uevent: version 1.0.3 [ 4.479607] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.208945] random: fast init done Starting dracut initqueue hook... [ 5.227896] virtio_net virtio0 ens2: renamed from eth0 [ 5.355234] scsi host0: ata_piix [ 5.356939] scsi host1: ata_piix [ 5.359019] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.361677] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.791132] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.048222] random: crng init done [ 10.049748] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.561949] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.799137] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.055755] SELinux: Disabled at runtime. [ 12.118059] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.125678] systemd[1]: Detected virtualization kvm. [ 12.127211] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.608498] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.612742] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.618221] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.622452] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.625893] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.634421] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.640328] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ 12.742527] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.079585] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.407066] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.430821] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.577488] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.599040] EDAC sbridge: Ver: 1.1.2 [ 14.987682] Key type dns_resolver registered [ 15.294952] NFS: Registering the id_resolver key type [ 15.296519] Key type id_resolver registered [ 15.297502] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg218-server login: [ 45.154977] hrtimer: interrupt took 4657731 ns [ 66.749170] libcfs: loading out-of-tree module taints kernel. [ 66.857649] Key type ._llcrypt registered [ 66.860062] Key type .llcrypt registered [ 67.085197] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 87.774800] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 89.853197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 89.901627] alg: No test for adler32 (adler32-zlib) [ 91.721912] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 93.094975] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 94.903167] Key type lgssc registered [ 97.447407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 121.164160] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 168.446332] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 182.057919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 182.109239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.633250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 183.684642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 183.832205] Lustre: lustre-MDT0000: new disk, initializing [ 183.946755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.972978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 188.601361] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.052469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.163984] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 203.188158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 203.192991] Lustre: Skipped 1 previous similar message [ 203.328345] Lustre: lustre-MDT0001: new disk, initializing [ 203.389657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.410511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 203.439063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 208.500795] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.739094] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 225.988551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.488634] Lustre: lustre-OST0000: new disk, initializing [ 226.494159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 226.499522] Lustre: 8456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.579408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.005240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 228.014518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 228.054493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 233.764831] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.095327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.404834] Lustre: lustre-OST0001: new disk, initializing [ 251.436945] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 251.448844] Lustre: 9529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.578554] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 258.157625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 258.183606] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 258.303034] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 258.753461] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.321920] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 277.672798] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 284.081592] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing check_logdir /tmp/testlogs/ [ 289.844329] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing yml_node [ 294.217287] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 296.640983] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 299.124280] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 301.804733] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 16:38:52 EDT 2026 [ 321.217239] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 323.652928] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 341.983979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 341.991152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.007347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.570248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.571235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.612498] Lustre: Skipped 2 previous similar messages [ 350.697718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.710923] Lustre: Skipped 3 previous similar messages [ 355.296810] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 355.526739] Lustre: server umount lustre-MDT0000 complete [ 360.141405] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640793 with bad export cookie 13644601461183782062 [ 360.149967] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 360.163162] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 360.934579] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.936273] LustreError: 8464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.951597] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 360.967608] Lustre: Skipped 3 previous similar messages [ 360.994721] LustreError: 8464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 366.048423] LustreError: 6527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.049623] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.097875] Lustre: Skipped 2 previous similar messages [ 366.642638] Lustre: server umount lustre-MDT0001 complete [ 371.821124] Lustre: server umount lustre-OST0000 complete [ 376.814977] Lustre: server umount lustre-OST0001 complete [ 386.487756] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 397.222573] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 429.981631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.257293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 430.287812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 430.382871] Lustre: lustre-MDT0000: new disk, initializing [ 430.453247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.477210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 435.521289] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 446.302513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.513339] Lustre: 15084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 446.573788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 446.577764] Lustre: Skipped 1 previous similar message [ 446.643315] Lustre: lustre-MDT0001: new disk, initializing [ 446.751863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 446.794255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 446.804335] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 452.059989] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 461.643890] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 468.917183] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 477.416389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 477.735177] Lustre: lustre-OST0000: new disk, initializing [ 477.738351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 477.742356] Lustre: 17054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 477.827144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.328314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 479.342147] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 479.638234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 484.772303] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 494.449150] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 499.588626] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 499.997369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 506.138669] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 506.456228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 510.434134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.443806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.473448] LustreError: Skipped 1 previous similar message [ 510.480453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.534965] Lustre: Skipped 1 previous similar message [ 523.235510] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 523.455544] Lustre: server umount lustre-OST0000 complete [ 528.866882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 528.870939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.907831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 528.924420] Lustre: Skipped 5 previous similar messages [ 533.877336] Lustre: server umount lustre-MDT0000 complete [ 536.038731] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.089743] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 538.252280] LustreError: 17053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788640971 with bad export cookie 13644601461183783602 [ 538.261164] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.270660] LustreError: 17053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 538.725788] Lustre: server umount lustre-MDT0001 complete [ 543.760735] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:42:54 (1788640974) [ 547.308988] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 549.872698] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:43:00 (1788640980) [ 559.066237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.753251] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.854844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 565.218500] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.255674] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 574.165096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.244705] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 588.258417] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 593.589440] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 601.353862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 608.013555] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 616.396763] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 629.731465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 629.737865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.755353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 629.775693] Lustre: Skipped 2 previous similar messages [ 629.795032] Lustre: Skipped 3 previous similar messages [ 634.267058] Lustre: server umount lustre-OST0000 complete [ 644.554955] Lustre: server umount lustre-MDT0000 complete [ 645.091497] LustreError: 19459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 645.112455] LustreError: 19459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 649.037281] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641081 with bad export cookie 13644601461183784484 [ 649.040894] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 649.057635] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 649.331529] Lustre: server umount lustre-MDT0001 complete [ 659.977742] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 663.024037] Key type lgssc unregistered [ 663.490644] LNet: 22998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.497159] LNetError: 22998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.511333] LNet: Removed LNI 192.168.202.118@tcp [ 664.895156] Key type .llcrypt unregistered [ 664.896895] Key type ._llcrypt unregistered [ 691.166429] Key type ._llcrypt registered [ 691.170141] Key type .llcrypt registered [ 691.365442] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 693.338835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 693.407026] alg: No test for adler32 (adler32-zlib) [ 694.404583] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 694.584715] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 696.247183] Key type lgssc registered [ 697.555742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 705.667637] loop: module loaded [ 705.749149] loop0: detected capacity change from 0 to 204800000 [ 705.894730] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 705.902314] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 706.127030] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 711.992351] loop0: detected capacity change from 0 to 204800000 [ 712.203759] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 712.217915] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 712.236109] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 716.767106] loop0: detected capacity change from 0 to 204800000 [ 722.104696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 722.126873] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.350906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 723.409842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 723.462461] Lustre: lustre-MDT0000: new disk, initializing [ 723.524954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 723.539919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 727.570722] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 735.462666] loop1: detected capacity change from 0 to 204800000 [ 741.128278] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 741.517631] Lustre: lustre-OST07c6: new disk, initializing [ 741.522296] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 741.525202] Lustre: Skipped 1 previous similar message [ 741.529852] Lustre: 26185:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 741.606689] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 747.595644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 747.614522] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 747.661584] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 747.752891] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 753.775529] Lustre: 25171:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 753.785653] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 753.798738] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 753.806532] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 753.812661] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 753.825147] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 767.668595] Lustre: 25171:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 767.684557] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 767.697520] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 767.715373] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 767.723821] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 767.730667] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.810586] Lustre: 25173:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 772.824573] Lustre: 25173:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 772.830578] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 772.837189] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.841783] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.847543] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.852834] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 772.858893] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.863400] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 772.872096] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.885470] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.907347] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.223972] Lustre: 25173:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 776.247068] Lustre: 25173:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 776.266121] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 776.278522] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.291436] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 776.315444] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.325198] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 776.331614] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.340687] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 776.352432] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.359559] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 776.366667] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.246102] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 782.259305] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 782.263456] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 782.267635] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.272097] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 782.278284] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.283683] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 782.295086] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.305312] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 782.320016] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.330819] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 782.342832] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.476237] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 790.486725] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 790.493158] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 790.498095] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 790.502675] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 790.506458] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 790.510303] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 790.519914] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 790.529606] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 790.536311] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 790.544165] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 790.548604] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.672532] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 806.678085] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 806.682459] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 806.692787] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 806.696671] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 806.699629] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 806.702898] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 806.706067] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 806.709556] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 806.711940] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 806.714904] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 806.717685] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 844.768990] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 844.782341] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.806214] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 848.354683] Lustre: server umount lustre-OST07c6 complete [ 858.374581] Lustre: server umount lustre-MDT0000 complete [ 878.009972] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 881.722765] Key type lgssc unregistered [ 882.101745] LNet: 28740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.115161] LNetError: 28740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.141540] LNet: Removed LNI 192.168.202.118@tcp [ 883.391734] Key type .llcrypt unregistered [ 883.393795] Key type ._llcrypt unregistered [ 907.475317] Key type ._llcrypt registered [ 907.480802] Key type .llcrypt registered [ 907.666482] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 909.833163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 909.999628] alg: No test for adler32 (adler32-zlib) [ 911.463173] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 911.756517] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 913.400125] Key type lgssc registered [ 915.108715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 925.683944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 925.724696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.434975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.469800] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 937.952880] LustreError: 30011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.982923] LustreError: 30011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 942.787201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.072222] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 943.339719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 950.704962] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 961.760931] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 970.882661] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 980.200075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 980.595393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 987.234253] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 996.231194] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 998.030729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1014.754353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1014.769475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.786121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1018.849924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.858650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1018.875429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1018.883241] Lustre: Skipped 1 previous similar message [ 1020.242511] Lustre: server umount lustre-OST0000 complete [ 1025.514944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1025.521633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.544561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.091368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1030.834751] Lustre: server umount lustre-MDT0000 complete [ 1034.210294] LustreError: 30933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1034.599317] LustreError: 30747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641467 with bad export cookie 902638947852319509 [ 1034.603517] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1034.606743] LustreError: 30747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1034.974769] Lustre: server umount lustre-MDT0001 complete [ 1046.392674] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1049.406370] Key type lgssc unregistered [ 1049.749671] LNet: 33750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.765497] LNetError: 33750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.788864] LNet: Removed LNI 192.168.202.118@tcp [ 1050.863273] Key type .llcrypt unregistered [ 1050.864947] Key type ._llcrypt unregistered [ 1071.823764] Key type ._llcrypt registered [ 1071.827205] Key type .llcrypt registered [ 1071.946829] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1073.047522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1073.266249] alg: No test for adler32 (adler32-zlib) [ 1074.332608] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 1074.662675] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1076.359236] Key type lgssc registered [ 1077.933516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1087.206096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1087.232213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.929001] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.982610] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1098.214453] LustreError: 35020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.261088] LustreError: 35020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1103.347753] LustreError: 35021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.204138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1106.814506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1113.143787] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1121.922911] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1128.703899] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1137.616527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1137.979523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1144.388844] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1154.190943] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1158.460645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1171.948719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1171.965435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1171.985350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1173.987888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1173.990824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1174.010620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.001590] Lustre: server umount lustre-OST0000 complete [ 1183.717172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1183.727597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.753719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.759934] Lustre: Skipped 1 previous similar message [ 1187.392945] Lustre: server umount lustre-MDT0000 complete [ 1189.350918] LustreError: 35943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.382484] LustreError: 35943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1192.094771] LustreError: 37830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641625 with bad export cookie 14064855117508620930 [ 1192.102474] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1192.109984] LustreError: 37830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1192.546634] Lustre: server umount lustre-MDT0001 complete [ 1205.075700] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1208.957373] Key type lgssc unregistered [ 1209.276582] LNet: 38582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.284293] LNetError: 38582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.314972] LNet: Removed LNI 192.168.202.118@tcp [ 1210.707567] Key type .llcrypt unregistered [ 1210.708890] Key type ._llcrypt unregistered [ 1239.036603] Key type ._llcrypt registered [ 1239.038704] Key type .llcrypt registered [ 1239.204174] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1240.539183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1240.553992] alg: No test for adler32 (adler32-zlib) [ 1241.730275] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 1241.979880] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1243.783829] Key type lgssc registered [ 1244.979494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1254.821129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1254.842270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.410283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1261.978084] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1266.661274] LustreError: 39994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.689838] LustreError: 39994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1271.783331] LustreError: 39993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.204809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.553464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1279.344122] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1287.488362] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1293.635763] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1302.152903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1302.718346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1311.163627] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1321.038199] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1326.345673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1331.689824] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.706687] Lustre: Skipped 1 previous similar message [ 1331.710403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1331.730235] Lustre: Skipped 2 previous similar messages [ 1332.484691] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 1334.057707] Lustre: server umount lustre-MDT0000 complete [ 1336.808370] LustreError: 39994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.827730] LustreError: 39994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1338.479878] LustreError: 39975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641771 with bad export cookie 2301353074255505036 [ 1338.484870] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1338.492452] LustreError: 39975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1341.922741] LustreError: 39990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.944965] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.950389] LustreError: 39990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1341.987643] Lustre: Skipped 1 previous similar message [ 1342.003300] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1345.135688] Lustre: server umount lustre-MDT0001 complete [ 1368.037736] Lustre: server umount lustre-OST0000 complete [ 1386.578370] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1389.670545] Key type lgssc unregistered [ 1389.977403] LNet: 43857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.984606] LNetError: 43857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.019530] LNet: Removed LNI 192.168.202.118@tcp [ 1391.116996] Key type .llcrypt unregistered [ 1391.120856] Key type ._llcrypt unregistered [ 1419.287751] Key type ._llcrypt registered [ 1419.290241] Key type .llcrypt registered [ 1419.394552] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1420.634722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1420.845457] alg: No test for adler32 (adler32-zlib) [ 1421.909835] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 1422.101840] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1423.775311] Key type lgssc registered [ 1424.959855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1433.510467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1433.552869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1435.119996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.666185] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1444.328297] LustreError: 45273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1444.355183] LustreError: 45273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1449.452021] LustreError: 45273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1451.326541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.809811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1457.895233] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1467.998845] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1475.890765] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1484.111877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1484.527871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1493.949269] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1504.086606] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1505.127430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1515.239689] Lustre: lustre-OST0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 1515.492991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1515.493710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.523987] Lustre: Skipped 1 previous similar message [ 1520.504691] LustreError: 47098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1520.633533] Lustre: server umount lustre-OST0000 complete [ 1541.092093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.097076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1541.149889] Lustre: Skipped 1 previous similar message [ 1541.164884] Lustre: Skipped 2 previous similar messages [ 1546.211568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1546.226379] Lustre: Skipped 2 previous similar messages [ 1551.332947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1551.340058] Lustre: Skipped 1 previous similar message [ 1554.399317] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1554.629050] Lustre: server umount lustre-MDT0000 complete [ 1555.426583] LustreError: 45268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1555.437737] LustreError: 45268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1558.586804] LustreError: 45254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788641991 with bad export cookie 4914279466423720578 [ 1558.592135] LustreError: 45254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1558.592654] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1565.093762] Lustre: server umount lustre-MDT0001 complete [ 1576.360988] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1579.228096] Key type lgssc unregistered [ 1579.652151] LNet: 48982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.668737] LNetError: 48982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.695901] LNet: Removed LNI 192.168.202.118@tcp [ 1580.603750] Key type .llcrypt unregistered [ 1580.609727] Key type ._llcrypt unregistered [ 1607.300693] Key type ._llcrypt registered [ 1607.302957] Key type .llcrypt registered [ 1607.436294] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1609.396149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1609.471541] alg: No test for adler32 (adler32-zlib) [ 1610.627622] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 1610.883219] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1612.607780] Key type lgssc registered [ 1613.837848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1621.505937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1621.522971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1623.027490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.825865] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1632.234400] LustreError: 50398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1632.268153] LustreError: 50398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1637.344774] LustreError: 50397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1639.206266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1639.661581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1646.074956] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1657.128627] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1663.860687] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1670.813422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1671.105518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1678.060114] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1685.397977] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1692.609344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1702.193773] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1705.998914] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1718.504667] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0517h02m28s [ 1720.284366] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1723.140784] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 1723.361036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.403649] Lustre: Skipped 1 previous similar message [ 1724.175569] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 1724.184749] Lustre: Skipped 3 previous similar messages [ 1729.004472] Lustre: server umount lustre-MDT0000 complete [ 1729.222634] LustreError: 50392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.098835] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1735.361164] LustreError: 52855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1735.375822] LustreError: 52855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1741.362900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.502923] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1741.504201] LustreError: 50394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.543741] LustreError: 50394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1741.733097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1741.793665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1742.786491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1746.938449] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1750.925932] Lustre: DEBUG MARKER: Wait for df (36694) ... [ 1752.654075] Lustre: DEBUG MARKER: done [ 1757.151903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1757.159868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.167331] Lustre: Skipped 1 previous similar message [ 1757.173197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1757.176082] Lustre: Skipped 3 previous similar messages [ 1758.184322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1763.297295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1763.305063] Lustre: Skipped 3 previous similar messages [ 1768.428181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1770.975180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1771.163924] Lustre: server umount lustre-OST0000 complete [ 1778.658927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1778.659380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1778.665831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1778.665839] Lustre: Skipped 1 previous similar message [ 1778.708226] Lustre: Skipped 1 previous similar message [ 1782.084063] Lustre: server umount lustre-MDT0000 complete [ 1785.833222] LustreError: 51150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.842118] LustreError: 51150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1786.010282] LustreError: 50377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642218 with bad export cookie 817712426069689348 [ 1786.013199] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1786.021180] LustreError: 50377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1786.515229] Lustre: server umount lustre-MDT0001 complete [ 1797.921890] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 1801.037870] Key type lgssc unregistered [ 1801.472475] LNet: 55359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.482381] LNetError: 55359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.528815] LNet: Removed LNI 192.168.202.118@tcp [ 1802.850715] Key type .llcrypt unregistered [ 1802.855721] Key type ._llcrypt unregistered [ 1848.381940] Key type ._llcrypt registered [ 1848.385521] Key type .llcrypt registered [ 1848.527959] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 1850.053740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.086733] alg: No test for adler32 (adler32-zlib) [ 1851.187396] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 1851.443898] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 1853.127583] Key type lgssc registered [ 1854.512924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1863.777331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1863.811979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.038686] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1865.052578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1865.169358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.656931] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1882.546223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1882.756765] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1882.800795] Lustre: 57565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1883.084462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1888.803699] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1897.583632] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1905.345579] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1913.212059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1913.479076] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1913.716951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1920.251788] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 1923.191957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1928.959784] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1935.417352] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0517h06m06s [ 1937.240368] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1945.733517] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1949.065340] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1949.160779] LustreError: 57569:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1969.399149] Lustre: lustre-MDT0000: Client b2770204-bc58-4f7a-b305-ed10a0907426 (at 192.168.202.18@tcp) reconnecting [ 1989.255461] LustreError: 57569:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1991.387518] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1993.756657] Lustre: DEBUG MARKER: done [ 2004.961195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2004.976739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.987355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2005.476610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2005.492300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.515368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2005.521902] Lustre: Skipped 1 previous similar message [ 2008.320987] Lustre: server umount lustre-OST0000 complete [ 2015.715115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.743789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.226891] Lustre: server umount lustre-MDT0000 complete [ 2023.652064] LustreError: 58306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642456 with bad export cookie 15351028319043345998 [ 2023.658175] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2023.671996] LustreError: 58306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2024.356608] Lustre: server umount lustre-MDT0001 complete [ 2035.058682] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2037.682791] Key type lgssc unregistered [ 2037.988486] LNet: 61895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.994567] LNetError: 61895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.017902] LNet: Removed LNI 192.168.202.118@tcp [ 2038.804912] Key type .llcrypt unregistered [ 2038.806122] Key type ._llcrypt unregistered [ 2079.428751] Key type ._llcrypt registered [ 2079.430862] Key type .llcrypt registered [ 2079.513862] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2080.309728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2080.464038] alg: No test for adler32 (adler32-zlib) [ 2081.503135] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 2081.687398] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2083.335486] Key type lgssc registered [ 2084.295802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2087.866763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2087.882589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.272271] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2089.290248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2089.468114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.598762] Lustre: Failing over lustre-MDT0000 [ 2091.039291] Lustre: server umount lustre-MDT0000 complete [ 2096.577422] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:08:48 (1788642528) [ 2103.053470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.608891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.359391] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2117.861777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.012916] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2118.065186] Lustre: 64679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2118.357973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2123.436658] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2131.265810] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2137.422144] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2146.028036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2146.293949] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2146.500060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.620356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2153.831825] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2161.951286] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2167.613751] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2174.394264] Lustre: server umount lustre-MDT0000 complete [ 2174.446848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2174.479057] Lustre: Skipped 1 previous similar message [ 2178.047786] LustreError: 64672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642611 with bad export cookie 14200443129635343137 [ 2178.051248] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2178.056845] LustreError: 64672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2178.680751] Lustre: server umount lustre-MDT0001 complete [ 2182.440639] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2188.056296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2195.392219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.743174] Lustre: 63605:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788642612/real 1788642612] req@00000000510f67db x1875527612799360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788642628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2195.769692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2195.780762] Lustre: Skipped 1 previous similar message [ 2199.967341] Lustre: 63604:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788642616/real 1788642616] req@00000000c6da1455 x1875527612799488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788642632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2200.991345] Lustre: 63603:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788642617/real 1788642617] req@000000009e9fed4c x1875527612799616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788642633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2204.000064] LustreError: 63602:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000075d42151 x1875527612800512/t0(0) o250->MGC192.168.202.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2204.022126] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2204.042028] Lustre: MGS: Client 7fc3b412-7ad6-4a19-8e3d-96135da502d8 (at 0@lo) reconnecting [ 2204.052117] Lustre: MGC192.168.202.118@tcp: Connection restored to 0@lo (at 0@lo) [ 2204.111294] Lustre: 63606:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788642621/real 1788642621] req@000000005f7aefa7 x1875527612799872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788642637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2204.466082] LustreError: 66530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.493043] LustreError: 66530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2204.585965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2204.660809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2205.602112] LustreError: 68451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.622367] LustreError: 68451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2209.565118] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2209.695115] Lustre: 63606:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788642626/real 1788642626] req@000000009191b02b x1875527612800000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788642642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2210.788329] LustreError: 68450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.822120] LustreError: 68450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2214.888925] LustreError: 66531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.892547] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2214.899633] LustreError: 66531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2217.697557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.016418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2222.369552] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2223.085149] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2229.655570] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2235.427219] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2242.577522] Lustre: server umount lustre-MDT0000 complete [ 2243.555981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2243.559639] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.582777] LustreError: 66531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.595024] Lustre: Skipped 2 previous similar messages [ 2243.631241] LustreError: 66531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2246.602361] LustreError: 64674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642679 with bad export cookie 14200443129635345412 [ 2246.613859] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2246.620595] LustreError: 64674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2247.281890] Lustre: server umount lustre-MDT0001 complete [ 2255.754415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2265.055926] Lustre: 63606:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788642681/real 1788642681] req@000000003bf85f23 x1875527612826112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788642697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2265.116085] Lustre: 63606:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2265.136424] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2267.163586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.666479] LustreError: 69185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.759503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.820061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2276.260248] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2283.747101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2285.071055] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2288.607180] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2297.716220] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2303.183310] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2310.070454] Lustre: server umount lustre-MDT0000 complete [ 2310.625571] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.640296] LustreError: 69185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.651104] Lustre: Skipped 2 previous similar messages [ 2310.683719] LustreError: 69185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2314.320525] LustreError: 67391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642747 with bad export cookie 14200443129635346798 [ 2314.333544] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2314.806989] Lustre: server umount lustre-MDT0001 complete [ 2322.199590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2325.712059] Lustre: DEBUG MARKER: files compared the same [ 2337.940402] Lustre: server umount lustre-OST0000 complete [ 2356.019801] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2359.269739] Key type lgssc unregistered [ 2359.552379] LNet: 74407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2359.561643] LNetError: 74407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2359.581870] LNet: Removed LNI 192.168.202.118@tcp [ 2360.502289] Key type .llcrypt unregistered [ 2360.506357] Key type ._llcrypt unregistered [ 2384.237216] Key type ._llcrypt registered [ 2384.238677] Key type .llcrypt registered [ 2384.339296] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2385.674279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2385.853010] alg: No test for adler32 (adler32-zlib) [ 2387.031412] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 2387.331210] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2389.127632] Key type lgssc registered [ 2390.629077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2398.138952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2398.155330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2399.797902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.189419] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug +malloc all [ 2409.956745] LustreError: 75820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.976240] LustreError: 75820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2413.529131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2413.828490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2418.517797] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug +malloc all [ 2426.300766] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2432.336350] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2439.714093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2440.127738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2444.223416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2446.795349] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug +malloc all [ 2454.266222] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2468.838968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.846735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2468.860433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2473.247817] Lustre: server umount lustre-OST0000 complete [ 2479.071855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.087818] Lustre: Skipped 1 previous similar message [ 2479.107907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2479.117853] Lustre: Skipped 2 previous similar messages [ 2483.852555] Lustre: server umount lustre-MDT0000 complete [ 2484.198065] LustreError: 75820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.237815] LustreError: 75820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2487.682604] LustreError: 75800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642920 with bad export cookie 12457353332722062419 [ 2487.688120] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2487.696818] LustreError: 75800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2488.034527] Lustre: server umount lustre-MDT0001 complete [ 2500.416196] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2504.094575] Key type lgssc unregistered [ 2504.457674] LNet: 79382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2504.473800] LNetError: 79382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2504.509322] LNet: Removed LNI 192.168.202.118@tcp [ 2505.531903] Key type .llcrypt unregistered [ 2505.533929] Key type ._llcrypt unregistered [ 2531.686197] Key type ._llcrypt registered [ 2531.689614] Key type .llcrypt registered [ 2531.807942] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2532.796282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2532.943943] alg: No test for adler32 (adler32-zlib) [ 2534.091608] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 2534.376858] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2536.031178] Key type lgssc registered [ 2537.421744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2545.074844] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2545.121456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2560.800857] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2565.984679] LustreError: 80772:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.118@tcp: failed processing log, type 4: rc = -110 [ 2592.607709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.496281] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2608.191806] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2616.171421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.410474] Lustre: *** cfs_fail_loc=706, val=0*** [ 2616.862202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.952968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2621.924677] LustreError: 80797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.932294] LustreError: 80797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2622.358576] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2627.044784] LustreError: 80798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.068694] LustreError: 81857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.100129] LustreError: 81857:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2630.840504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.324245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2635.867516] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2643.640648] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2650.946904] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2656.737554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2656.746352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2656.756411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2657.252618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.275941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2657.281390] Lustre: Skipped 1 previous similar message [ 2659.487132] Lustre: server umount lustre-OST0000 complete [ 2666.977986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2666.985882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2667.010769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2669.341853] Lustre: server umount lustre-MDT0000 complete [ 2672.610924] LustreError: 81862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.620609] LustreError: 81862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2673.242862] LustreError: 80778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643106 with bad export cookie 9065153271803155452 [ 2673.250705] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2673.262653] LustreError: 80778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2673.611236] Lustre: server umount lustre-MDT0001 complete [ 2684.178615] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 2687.293305] Key type lgssc unregistered [ 2687.618850] LNet: 84386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2687.629421] LNetError: 84386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2688.678047] LNet: Removed LNI 192.168.202.118@tcp [ 2689.632405] Key type .llcrypt unregistered [ 2689.637077] Key type ._llcrypt unregistered [ 2713.405922] Key type ._llcrypt registered [ 2713.408318] Key type .llcrypt registered [ 2713.497835] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2714.339652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2714.370839] alg: No test for adler32 (adler32-zlib) [ 2715.502145] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 2715.713760] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 2717.359279] Key type lgssc registered [ 2718.734410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2727.252203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2727.279841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2733.529963] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2742.947088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.796463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2748.896169] LustreError: 86487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.908408] LustreError: 86487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2749.220339] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2754.016726] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.073878] LustreError: 86485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2759.015828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2759.142953] LustreError: 86487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2759.179431] LustreError: 86487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2759.475822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2764.782286] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.810826] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2766.550716] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2768.878150] LustreError: 86490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.926330] LustreError: 86490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2776.197268] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2776.210034] Lustre: MGS: Client f6f3fbc8-259c-40a5-98cc-ca01cc534171 (at 0@lo) reconnecting [ 2776.256952] LustreError: 86501:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2776.677697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.818254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2781.815186] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2797.033736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2797.040385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2797.063936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2797.075212] Lustre: Skipped 1 previous similar message [ 2797.096254] Lustre: Skipped 1 previous similar message [ 2800.397864] Lustre: server umount lustre-OST0000 complete [ 2807.270971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2807.278268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2807.288408] Lustre: Skipped 1 previous similar message [ 2807.303943] Lustre: Skipped 1 previous similar message [ 2812.386226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2812.399168] Lustre: Skipped 1 previous similar message [ 2817.537079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2817.540848] Lustre: Skipped 1 previous similar message [ 2819.039399] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2819.377657] Lustre: server umount lustre-MDT0000 complete [ 2822.634863] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.675598] LustreError: 86486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2824.537409] Lustre: server umount lustre-MDT0001 complete [ 2829.131897] Lustre: server umount MGS complete [ 2843.047659] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:21:13 (1788643273) [ 2874.309327] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 2883.992371] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 2919.396952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2924.327711] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2933.701676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.976395] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2934.017720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2934.082512] Lustre: lustre-MDT0001: new disk, initializing [ 2934.151535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2934.172715] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2934.183360] Lustre: 93658:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2934.196449] LustreError: 93658:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2938.363188] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2948.236882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2948.511809] Lustre: lustre-OST0000: new disk, initializing [ 2948.519062] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2948.527098] Lustre: 94423:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2948.594084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2954.045902] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2958.948676] Lustre: 94923:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000840f661 x1875528277598336/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788643407 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2963.883277] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2971.093948] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2971.120114] Lustre: MGS: Client 9678ec06-c9b0-4230-809c-17da4d12d2e9 (at 0@lo) reconnecting [ 2971.190118] Lustre: 92926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2971.224458] LustreError: 93644:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2971.289298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2971.438796] Lustre: lustre-MDT0000: new disk, initializing [ 2971.574439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2971.632394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2971.810691] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2972.710852] Lustre: 95496:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000076408206 x1875528277604864/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788643421 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2977.170606] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 2977.795467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2977.812633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2978.043768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3002.861064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3002.882961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3002.891520] Lustre: Skipped 1 previous similar message [ 3013.093579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3013.107668] Lustre: Skipped 3 previous similar messages [ 3016.159186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3016.338209] Lustre: server umount lustre-OST0000 complete [ 3023.332888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3023.338146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3023.354505] Lustre: Skipped 1 previous similar message [ 3025.512270] Lustre: server umount lustre-MDT0000 complete [ 3028.452605] LustreError: 93634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3028.467502] LustreError: 93634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3035.690630] Lustre: server umount lustre-MDT0001 complete [ 3039.379805] Lustre: server umount MGS complete [ 3050.031540] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:24:41 (1788643481) [ 3066.771194] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3069.841706] Key type lgssc unregistered [ 3070.153187] LNet: 97992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3070.161453] LNetError: 97992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3070.189063] LNet: Removed LNI 192.168.202.118@tcp [ 3071.258236] Key type .llcrypt unregistered [ 3071.260524] Key type ._llcrypt unregistered [ 3089.838619] Key type ._llcrypt registered [ 3089.845786] Key type .llcrypt registered [ 3089.971303] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3091.355886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3091.375586] alg: No test for adler32 (adler32-zlib) [ 3092.554782] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 3092.920054] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3094.640088] Key type lgssc registered [ 3096.046363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3104.357445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3104.454835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.974687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3111.147798] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3115.490109] LustreError: 99210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3115.512796] LustreError: 99210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3119.988698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.282661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3124.244558] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3130.441588] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3137.023725] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3144.442685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3144.763493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.906633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3152.134398] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3161.213446] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3171.301286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3171.308966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.314199] LustreError: Skipped 1 previous similar message [ 3171.317442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3171.327627] Lustre: Skipped 1 previous similar message [ 3171.339226] Lustre: Skipped 1 previous similar message [ 3176.422509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3176.429320] Lustre: Skipped 1 previous similar message [ 3176.844747] Lustre: server umount lustre-OST0000 complete [ 3181.542127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3181.558600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3181.560912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3181.600645] Lustre: Skipped 1 previous similar message [ 3186.663550] LustreError: 99207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3186.682155] LustreError: 99207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3186.754635] Lustre: server umount lustre-MDT0000 complete [ 3190.661780] LustreError: 99190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643623 with bad export cookie 4460368818149324275 [ 3190.669383] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3190.677856] LustreError: 99190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3191.204552] Lustre: server umount lustre-MDT0001 complete [ 3204.155315] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3206.440683] Key type lgssc unregistered [ 3206.710612] LNet: 102744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3206.715893] LNetError: 102744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3206.726961] LNet: Removed LNI 192.168.202.118@tcp [ 3207.529504] Key type .llcrypt unregistered [ 3207.532533] Key type ._llcrypt unregistered [ 3223.254983] Key type ._llcrypt registered [ 3223.258566] Key type .llcrypt registered [ 3223.417039] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3224.332812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3224.471031] alg: No test for adler32 (adler32-zlib) [ 3225.801573] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 3226.049163] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3227.817524] Key type lgssc registered [ 3229.288912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3235.663199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3235.669555] LustreError: 103798:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3235.715598] LustreError: 103805:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3235.720956] LustreError: 103798:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4952 [ 3235.728749] LustreError: 103798:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3235.738853] LustreError: 103798:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3235.740858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3237.389797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3244.929118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3245.253428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3250.488903] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3256.258075] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing lsmod [ 3259.538090] LustreError: 105001:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3259.613878] LustreError: 105012:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3259.626449] LustreError: 105001:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4919 [ 3259.642444] LustreError: 105001:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3259.654059] LustreError: 105001:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3259.671808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3260.119663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3262.198855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3266.017104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3266.022076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.032744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3267.552282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3267.555620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3267.565291] Lustre: Skipped 1 previous similar message [ 3267.588281] Lustre: Skipped 1 previous similar message [ 3270.701154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3270.707644] Lustre: Skipped 3 previous similar messages [ 3275.748942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3277.279175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3277.679217] Lustre: server umount lustre-MDT0000 complete [ 3280.872553] LustreError: 103838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3280.923385] LustreError: 103838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3282.604806] LustreError: 103821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643715 with bad export cookie 3491852615387322160 [ 3282.625026] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3283.067276] LustreError: 104274:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bbd81565 x1875528812353920/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3283.393546] Lustre: server umount lustre-MDT0001 complete [ 3297.830624] Lustre: server umount lustre-OST0000 complete [ 3306.415525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.869076] LustreError: 106076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3306.993552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3312.102059] LustreError: 106077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.209886] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3316.212329] LustreError: 106076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.807883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.095680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3325.339581] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3332.862779] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3338.885159] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3345.397122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3345.864524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3352.053751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3352.396076] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3360.603250] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3371.492305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3371.502553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3371.530473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3371.538749] Lustre: Skipped 2 previous similar messages [ 3372.515182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3374.571837] Lustre: server umount lustre-OST0000 complete [ 3382.752279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3382.756797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.768348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.786038] Lustre: Skipped 2 previous similar messages [ 3382.806974] Lustre: Skipped 3 previous similar messages [ 3385.467015] Lustre: server umount lustre-MDT0000 complete [ 3387.875890] LustreError: 106077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.890526] LustreError: 106077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3389.014162] LustreError: 106059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643821 with bad export cookie 3491852615387323077 [ 3389.018359] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3389.029646] LustreError: 106059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3389.247144] Lustre: server umount lustre-MDT0001 complete [ 3400.154799] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3403.381908] Key type lgssc unregistered [ 3403.625729] LNet: 109609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3403.630291] LNetError: 109609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3403.641872] LNet: Removed LNI 192.168.202.118@tcp [ 3404.633529] Key type .llcrypt unregistered [ 3404.641678] Key type ._llcrypt unregistered [ 3427.690798] Key type ._llcrypt registered [ 3427.693756] Key type .llcrypt registered [ 3427.771081] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3429.087073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3429.185719] alg: No test for adler32 (adler32-zlib) [ 3430.293470] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 3430.637444] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3432.375221] Key type lgssc registered [ 3433.631761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3440.649491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3440.670042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3441.981732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.654728] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3451.371129] LustreError: 111024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.394808] LustreError: 111024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3452.971790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.339354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3457.771742] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3465.264319] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3469.999728] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3475.880817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3476.266662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3482.325535] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3486.542428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3490.162784] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3496.547271] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3502.540768] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3509.216540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3509.227126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.269031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3512.290994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3512.298675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3512.300650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3512.340289] Lustre: Skipped 1 previous similar message [ 3513.783222] Lustre: server umount lustre-OST0000 complete [ 3519.968278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3519.988319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3520.027567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3522.530882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3522.553677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.443497] Lustre: server umount lustre-MDT0000 complete [ 3527.659208] LustreError: 111024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.678847] LustreError: 111024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3529.293038] LustreError: 111005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643962 with bad export cookie 181631526427825556 [ 3529.302385] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3529.311223] LustreError: 111005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3529.789398] Lustre: server umount lustre-MDT0001 complete [ 3540.654128] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3543.418635] Key type lgssc unregistered [ 3543.712501] LNet: 114663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3543.718033] LNetError: 114663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3543.728767] LNet: Removed LNI 192.168.202.118@tcp [ 3544.526485] Key type .llcrypt unregistered [ 3544.528744] Key type ._llcrypt unregistered [ 3562.567844] Key type ._llcrypt registered [ 3562.569594] Key type .llcrypt registered [ 3562.654445] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3563.653364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3563.785094] alg: No test for adler32 (adler32-zlib) [ 3564.875465] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 3565.068563] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3566.727177] Key type lgssc registered [ 3567.607889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3574.837879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3574.864479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.529185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.529537] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3587.041190] LustreError: 115881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3587.061617] LustreError: 115881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3590.505303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3590.876086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3595.107021] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3602.672722] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3609.653926] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3617.187367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3617.485364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3621.638109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3624.078931] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3632.014391] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3651.557958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3651.563143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3651.573396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3651.594672] Lustre: Skipped 1 previous similar message [ 3651.749941] Lustre: server umount lustre-OST0000 complete [ 3656.675771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3656.679569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3656.694145] Lustre: Skipped 2 previous similar messages [ 3656.703108] Lustre: Skipped 1 previous similar message [ 3661.796809] LustreError: 115876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.814872] LustreError: 115876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3661.907707] Lustre: server umount lustre-MDT0000 complete [ 3666.078526] LustreError: 115862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644099 with bad export cookie 3249485894452454313 [ 3666.094835] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3666.095621] LustreError: 115862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3666.646649] Lustre: server umount lustre-MDT0001 complete [ 3680.213469] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3686.426011] Key type lgssc unregistered [ 3687.025698] LNet: 119436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3687.042016] LNetError: 119436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3687.088905] LNet: Removed LNI 192.168.202.118@tcp [ 3688.940676] Key type .llcrypt unregistered [ 3688.942168] Key type ._llcrypt unregistered [ 3723.622356] Key type ._llcrypt registered [ 3723.625179] Key type .llcrypt registered [ 3723.727977] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3725.294708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3725.434391] alg: No test for adler32 (adler32-zlib) [ 3726.535825] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 3726.850200] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3728.567243] Key type lgssc registered [ 3729.782963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3738.218557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3738.240572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3739.745885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3744.509667] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3749.861180] LustreError: 120854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3749.890867] LustreError: 120854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3753.131767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3753.440872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3758.300469] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3766.682862] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3774.046818] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3782.545833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3782.882521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3784.961459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3790.024810] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3797.768964] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3805.383028] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3808.712842] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3808.719332] Lustre: Skipped 2 previous similar messages [ 3813.418493] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3813.425263] Lustre: Skipped 2 previous similar messages [ 3813.984520] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3814.028611] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3824.665960] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.18@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3824.676972] Lustre: Skipped 2 previous similar messages [ 3830.356509] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.18@tcp 192.168.0.[2,10]@tcp [ 3834.769789] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.18/24@tcp in log lustre-MDT0000 [ 3834.778443] Lustre: Skipped 2 previous similar messages [ 3836.836632] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.18/24@tcp [ 3836.849583] Lustre: Skipped 1 previous similar message [ 3844.594367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3844.597730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3844.617819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3844.639110] Lustre: Skipped 1 previous similar message [ 3849.696646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3849.701567] Lustre: Skipped 1 previous similar message [ 3854.818815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3854.842599] Lustre: Skipped 1 previous similar message [ 3858.399284] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3858.662857] Lustre: server umount lustre-OST0000 complete [ 3865.059800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3865.070882] Lustre: Skipped 1 previous similar message [ 3865.098038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3865.102108] Lustre: Skipped 1 previous similar message [ 3870.178209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3870.186885] Lustre: Skipped 1 previous similar message [ 3877.343505] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3877.804877] Lustre: server umount lustre-MDT0000 complete [ 3880.424901] LustreError: 120853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3880.440802] LustreError: 120853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3882.233081] LustreError: 120833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644315 with bad export cookie 15176960759924243867 [ 3882.238234] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3882.244974] LustreError: 120833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3882.779478] Lustre: server umount lustre-MDT0001 complete [ 3895.322392] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 3898.457947] Key type lgssc unregistered [ 3898.776565] LNet: 125595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3898.788034] LNetError: 125595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3898.836259] LNet: Removed LNI 192.168.202.118@tcp [ 3899.930194] Key type .llcrypt unregistered [ 3899.931870] Key type ._llcrypt unregistered [ 3928.664768] Key type ._llcrypt registered [ 3928.665922] Key type .llcrypt registered [ 3928.754243] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 3929.754146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3929.793643] alg: No test for adler32 (adler32-zlib) [ 3930.939211] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 3931.575094] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 3933.583232] Key type lgssc registered [ 3936.295346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3947.075766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3947.110669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3948.808736] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3948.875311] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.18/24@tcp [ 3948.891407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3952.828460] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3958.246546] LustreError: 127158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3958.263502] LustreError: 127158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3962.162724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3962.542631] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3962.598649] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.18/24@tcp [ 3962.735523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3967.551904] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 3977.264067] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3984.837940] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3993.192624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3993.506485] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3993.638137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3998.740832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4000.974150] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4009.154061] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4023.272229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4023.273342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4023.297424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4026.364995] Lustre: server umount lustre-OST0000 complete [ 4033.506405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4033.532783] Lustre: Skipped 1 previous similar message [ 4033.537201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4033.555591] Lustre: Skipped 2 previous similar messages [ 4037.538389] Lustre: server umount lustre-MDT0000 complete [ 4038.624934] LustreError: 129234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4038.669137] LustreError: 129234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4042.064907] LustreError: 127139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644475 with bad export cookie 65663892185486324 [ 4042.083937] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4042.517041] Lustre: server umount lustre-MDT0001 complete [ 4053.459529] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 4056.477712] Key type lgssc unregistered [ 4056.784765] LNet: 130745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4056.796353] LNetError: 130745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4056.817572] LNet: Removed LNI 192.168.202.118@tcp [ 4057.650818] Key type .llcrypt unregistered [ 4057.654884] Key type ._llcrypt unregistered [ 4086.342972] Key type ._llcrypt registered [ 4086.346116] Key type .llcrypt registered [ 4086.487734] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 4102.154439] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4103.355920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4103.374203] alg: No test for adler32 (adler32-zlib) [ 4104.705648] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 4104.911962] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 4106.567415] Key type lgssc registered [ 4107.947226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4143.382289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4143.416279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4144.728271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4144.778936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4144.841935] Lustre: lustre-MDT0000: new disk, initializing [ 4145.054247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4145.079386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4150.241531] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4161.472709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.648912] Lustre: 134142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4161.734399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4161.739529] Lustre: Skipped 1 previous similar message [ 4161.827764] Lustre: lustre-MDT0001: new disk, initializing [ 4161.919900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4161.955847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4161.972520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4166.798758] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4176.852227] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4182.782192] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4190.318884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4190.811990] Lustre: lustre-OST0000: new disk, initializing [ 4190.817763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4190.825094] Lustre: 136109:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4190.985733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4196.973875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4197.000588] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4197.137538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4198.953644] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4209.834189] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4215.256180] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4215.494717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4220.718513] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4220.955962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4222.944908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4222.947529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.958320] LustreError: Skipped 1 previous similar message [ 4222.968064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4222.971270] Lustre: Skipped 1 previous similar message [ 4222.987752] Lustre: Skipped 1 previous similar message [ 4228.073207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4228.083749] Lustre: Skipped 1 previous similar message [ 4233.186069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4233.198737] Lustre: Skipped 1 previous similar message [ 4237.279144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4237.418822] Lustre: server umount lustre-OST0000 complete [ 4243.427573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4243.460063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4243.468859] Lustre: Skipped 1 previous similar message [ 4247.577640] Lustre: server umount lustre-MDT0000 complete [ 4252.110716] LustreError: 134132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644685 with bad export cookie 17533613038008741163 [ 4252.119581] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4252.127363] LustreError: 134132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4252.575914] Lustre: server umount lustre-MDT0001 complete [ 4262.087246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4262.652727] LustreError: 138328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4262.675361] LustreError: 138328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4262.737845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4267.271835] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4268.002458] LustreError: 138329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4272.098312] LustreError: 138328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4277.217956] LustreError: 138329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4277.407583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4277.790797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4282.932531] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4290.770924] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4296.457318] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4303.175731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4303.550684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4310.204347] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4317.158120] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4321.923183] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4322.096781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4326.098830] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4326.374486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4335.401592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4335.641226] Lustre: lustre-OST0001: new disk, initializing [ 4335.647402] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4335.652711] Lustre: 141699:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4335.799201] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4341.845224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4341.864376] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4341.975753] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4342.572708] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4353.189313] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4353.534759] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4358.970472] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4359.255913] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4365.598326] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90d90d06f000.ost_server_uuid 50 [ 4366.909872] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90d90d06f000.ost_server_uuid in IDLE state after 0 sec [ 4378.081244] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4378.087286] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4378.099081] LustreError: Skipped 1 previous similar message [ 4378.100620] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4378.128309] Lustre: Skipped 2 previous similar messages [ 4382.004312] Lustre: server umount lustre-OST0001 complete [ 4383.202612] LustreError: 140147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.232336] LustreError: 140147:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4388.327951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.359069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4388.362772] Lustre: Skipped 2 previous similar messages [ 4392.722739] Lustre: server umount lustre-OST0000 complete [ 4398.560816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.574726] Lustre: Skipped 2 previous similar messages [ 4408.805343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4408.817768] Lustre: Skipped 5 previous similar messages [ 4411.359173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4411.704876] Lustre: server umount lustre-MDT0000 complete [ 4413.930224] LustreError: 143002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.982357] LustreError: 143002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4415.772966] LustreError: 138310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644848 with bad export cookie 17533613038008742052 [ 4415.776029] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4415.787583] LustreError: 138310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4416.235569] Lustre: server umount lustre-MDT0001 complete [ 4435.377925] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 4437.679537] Key type lgssc unregistered [ 4438.011491] LNet: 144704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4438.021827] LNetError: 144704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4438.040699] LNet: Removed LNI 192.168.202.118@tcp [ 4438.911531] Key type .llcrypt unregistered [ 4438.914376] Key type ._llcrypt unregistered [ 4483.263765] Key type ._llcrypt registered [ 4483.265649] Key type .llcrypt registered [ 4483.334743] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 4498.700835] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4500.452850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4500.493569] alg: No test for adler32 (adler32-zlib) [ 4501.803409] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 4502.145130] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 4503.952105] Key type lgssc registered [ 4505.069394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4543.637060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4543.664511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4544.968896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4545.002763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4545.070355] Lustre: lustre-MDT0000: new disk, initializing [ 4545.194580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4545.222947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4550.480842] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4561.508370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.695648] Lustre: 148854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4561.774161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4561.781478] Lustre: Skipped 1 previous similar message [ 4561.967044] Lustre: lustre-MDT0001: new disk, initializing [ 4562.146859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4562.222621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4562.240801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4567.629353] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4577.477638] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4585.424890] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4593.292517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4593.702493] Lustre: lustre-OST0000: new disk, initializing [ 4593.707537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4593.715392] Lustre: 150825:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4593.872308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4598.843349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4598.851711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4598.979603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4600.465519] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4610.676934] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4615.553398] Lustre: Failing over lustre-OST0000 [ 4615.744566] Lustre: server umount lustre-OST0000 complete [ 4633.056857] Lustre: 147077:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788645050/real 1788645050] req@000000002b150b1b x1875530151397248/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788645066 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4633.100574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.084172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4635.104104] Lustre: 147079:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788645052/real 1788645052] req@00000000635916ca x1875530151397504/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788645068 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4635.132620] Lustre: 147079:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4635.139976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.503581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4635.527868] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4640.223940] Lustre: 147078:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788645057/real 1788645057] req@00000000c55d9021 x1875530151399296/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788645073 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4641.783858] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4643.624109] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4645.386815] Lustre: Failing over lustre-MDT0000 [ 4645.393040] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:08 [ 4645.624462] Lustre: server umount lustre-MDT0000 complete [ 4646.433996] Lustre: 147079:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788645062/real 1788645062] req@00000000c7454652 x1875530151400448/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788645078 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4646.438315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4646.458391] Lustre: 147079:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4646.489903] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4646.491643] Lustre: Skipped 1 previous similar message [ 4646.495586] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4651.489054] LustreError: 149659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.510860] LustreError: 149659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4655.300162] LustreError: 148861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4656.609381] LustreError: 148867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4656.631034] LustreError: 148867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4660.430655] LustreError: 152740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4663.263163] Lustre: 147080:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788645079/real 1788645079] req@000000002fefc3fe x1875530151410176/t0(0) o400->MGC192.168.202.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788645095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4663.299423] Lustre: 147080:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4663.306099] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4665.577411] LustreError: 148862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4665.641032] LustreError: 148862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4666.318046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4673.926755] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4674.090369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4674.214789] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4675.271209] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4679.146190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4679.198586] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4679.279765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4681.380853] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4689.392441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.393160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4689.403697] Lustre: Skipped 2 previous similar messages [ 4689.431301] Lustre: Skipped 1 previous similar message [ 4694.500293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4694.513373] Lustre: Skipped 1 previous similar message [ 4694.733702] Lustre: server umount lustre-OST0000 complete [ 4699.630124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4699.633797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4699.635674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4699.658583] Lustre: Skipped 1 previous similar message [ 4704.741043] LustreError: 150043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4704.766707] LustreError: 150043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4704.988729] Lustre: server umount lustre-MDT0000 complete [ 4708.758094] LustreError: 148848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645141 with bad export cookie 16442068688749936233 [ 4708.766572] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4708.771041] LustreError: 148848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4709.210862] Lustre: server umount lustre-MDT0001 complete [ 4720.715762] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 4723.169862] Key type lgssc unregistered [ 4723.598817] LNet: 154482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4723.604793] LNetError: 154482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4723.620071] LNet: Removed LNI 192.168.202.118@tcp [ 4724.559403] Key type .llcrypt unregistered [ 4724.561761] Key type ._llcrypt unregistered [ 4759.201341] Key type ._llcrypt registered [ 4759.203655] Key type .llcrypt registered [ 4759.282719] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4760.239355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4760.283075] alg: No test for adler32 (adler32-zlib) [ 4761.214101] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 4761.404736] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 4763.047192] Key type lgssc registered [ 4764.148761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4771.032431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4771.044715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4772.419422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4776.484288] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4782.563919] LustreError: 155998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.577855] LustreError: 155998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4784.499878] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4787.695974] LustreError: 155999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4788.392987] LustreError: 156027:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4788.403330] LustreError: 156027:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4788.414482] LustreError: 156027:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4788.754226] Lustre: server umount lustre-MDT0000 complete [ 4803.347771] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 4811.857847] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 4848.513861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.885222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4848.932120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4849.039429] Lustre: lustre-MDT0000: new disk, initializing [ 4849.153718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4849.188726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4854.853680] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4866.103518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4866.164167] Lustre: 159333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4866.203748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4866.205661] Lustre: Skipped 1 previous similar message [ 4866.254767] Lustre: lustre-MDT0001: new disk, initializing [ 4866.311749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4866.334681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4866.344856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4870.214553] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4880.560732] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4887.064069] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4895.181902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4895.873072] Lustre: lustre-OST0000: new disk, initializing [ 4895.885318] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4895.909710] Lustre: 161303:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4896.095953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4905.253369] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 4905.617620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4905.641724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4905.776589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4915.752984] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4920.851314] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4923.328533] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 17:55:54 (1788645354) [ 5156.831478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5156.832476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5156.837746] Lustre: Skipped 2 previous similar messages [ 5156.844438] Lustre: Skipped 2 previous similar messages [ 5160.859690] Lustre: server umount lustre-MDT0000 complete [ 5161.957491] LustreError: 163066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5161.983428] LustreError: 163066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5165.462105] LustreError: 159324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645598 with bad export cookie 12728759487128479769 [ 5165.466226] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5165.475620] LustreError: 159324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5167.072845] LustreError: 159339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5167.073869] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5167.110123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5170.079977] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5171.882810] Lustre: server umount lustre-MDT0001 complete [ 5182.153547] Lustre: server umount lustre-OST0000 complete [ 5191.343752] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 5201.020790] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5237.463778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5237.746333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5237.786996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5237.879360] Lustre: lustre-MDT0000: new disk, initializing [ 5237.963839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5237.971235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5241.883972] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5252.276986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.396194] Lustre: 165978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5252.435749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5252.440988] Lustre: Skipped 1 previous similar message [ 5252.522712] Lustre: lustre-MDT0001: new disk, initializing [ 5252.608339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5252.660211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5252.678075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5257.259810] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5267.639257] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5273.299441] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5280.328309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5280.613471] Lustre: lustre-OST0000: new disk, initializing [ 5280.616437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5280.621828] Lustre: 167945:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5280.707669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5287.443432] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5290.620214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5290.629644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5290.701467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5297.321693] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5301.407397] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5301.582454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5306.440964] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5306.754029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5311.465931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5311.475056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5311.506618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5314.875872] Lustre: server umount lustre-OST0000 complete [ 5319.136281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5319.147714] LustreError: Skipped 1 previous similar message [ 5319.159239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5319.187421] Lustre: Skipped 1 previous similar message [ 5319.190179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5319.198235] Lustre: Skipped 1 previous similar message [ 5324.556095] Lustre: server umount lustre-MDT0000 complete [ 5326.818335] LustreError: 165983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5326.835327] LustreError: 165983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5328.143069] LustreError: 165969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645761 with bad export cookie 12728759487128638599 [ 5328.144420] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5328.152351] LustreError: 165969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5328.664300] Lustre: server umount lustre-MDT0001 complete [ 5336.193367] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:02:47 (1788645767) [ 5350.231783] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 5359.891603] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5401.225972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5401.687584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5401.709273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5401.840271] Lustre: lustre-MDT0000: new disk, initializing [ 5402.070134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5402.120593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5409.324819] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5422.879861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5423.173729] Lustre: 172701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5423.604096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5429.665944] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5443.362964] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5450.645124] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5458.128314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5458.618708] Lustre: lustre-OST0000: new disk, initializing [ 5458.628981] Lustre: Skipped 1 previous similar message [ 5458.639306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5458.650190] Lustre: Skipped 2 previous similar messages [ 5458.657473] Lustre: 174681:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5466.166446] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5468.211166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5468.216857] Lustre: Skipped 1 previous similar message [ 5468.219155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5468.275629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5476.836913] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5488.609934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5488.612955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5488.629420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5488.645326] Lustre: Skipped 2 previous similar messages [ 5488.666506] Lustre: Skipped 3 previous similar messages [ 5498.335793] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5498.545929] Lustre: server umount lustre-OST0000 complete [ 5503.969526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.980037] Lustre: Skipped 1 previous similar message [ 5509.090526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5509.108379] Lustre: Skipped 5 previous similar messages [ 5517.279313] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5517.465564] Lustre: server umount lustre-MDT0000 complete [ 5518.311434] LustreError: 172708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.345639] LustreError: 172708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5521.184024] LustreError: 172692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645954 with bad export cookie 12728759487128639481 [ 5521.193283] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5521.202912] LustreError: 172692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5521.871336] Lustre: server umount lustre-MDT0001 complete [ 5530.481737] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:06:01 (1788645961) [ 5546.227908] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 5554.865304] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5604.151126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5604.603812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5604.631929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5604.740371] Lustre: lustre-MDT0000: new disk, initializing [ 5604.899506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5604.910344] Lustre: Skipped 2 previous similar messages [ 5604.938409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5609.909259] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5621.302453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5621.506415] Lustre: 179011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5621.851059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5627.476378] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5637.060539] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5643.061047] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5650.092574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5650.487181] Lustre: 180985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5656.374630] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5657.158312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5657.315255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5665.621617] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5672.938215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5672.942669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5672.948389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5672.948398] Lustre: Skipped 2 previous similar messages [ 5672.965725] Lustre: Skipped 1 previous similar message [ 5678.323340] Lustre: server umount lustre-OST0000 complete [ 5683.175575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5683.192096] LustreError: Skipped 1 previous similar message [ 5688.748675] Lustre: server umount lustre-MDT0000 complete [ 5692.538297] LustreError: 179003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646125 with bad export cookie 12728759487128640566 [ 5692.563058] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5692.908826] Lustre: server umount lustre-MDT0001 complete [ 5703.457518] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 5706.037830] Key type lgssc unregistered [ 5706.391030] LNet: 182840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5706.410594] LNetError: 182840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5706.430458] LNet: Removed LNI 192.168.202.118@tcp [ 5707.398799] Key type .llcrypt unregistered [ 5707.401195] Key type ._llcrypt unregistered [ 5732.915818] Key type ._llcrypt registered [ 5732.918490] Key type .llcrypt registered [ 5733.073965] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5735.247324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5735.268318] alg: No test for adler32 (adler32-zlib) [ 5736.566512] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 5737.022963] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 5738.703247] Key type lgssc registered [ 5739.984379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5747.293162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5747.318733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5748.766269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5753.139580] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5758.945296] LustreError: 184256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5758.960827] LustreError: 184256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5760.510119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5761.088630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5765.274919] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5772.016212] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5777.552543] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5783.662929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5784.033292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5790.338793] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5793.294186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5798.314789] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5820.899940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5820.921464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5820.954636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5822.409712] Lustre: server umount lustre-OST0000 complete [ 5829.093198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5829.108896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.112080] Lustre: Skipped 1 previous similar message [ 5829.124563] Lustre: Skipped 1 previous similar message [ 5834.211484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5838.306456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5838.315521] Lustre: Skipped 2 previous similar messages [ 5842.400463] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5842.779983] Lustre: server umount lustre-MDT0000 complete [ 5843.425819] LustreError: 185284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.454332] LustreError: 185284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5847.077282] LustreError: 184238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646280 with bad export cookie 4030684655673946019 [ 5847.089634] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5847.092258] LustreError: 184238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5847.511334] Lustre: server umount lustre-MDT0001 complete [ 5859.823931] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 5863.287412] Key type lgssc unregistered [ 5863.707772] LNet: 187813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5863.718982] LNetError: 187813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5864.765429] LNet: Removed LNI 192.168.202.118@tcp [ 5865.971206] Key type .llcrypt unregistered [ 5865.976876] Key type ._llcrypt unregistered [ 5893.380414] Key type ._llcrypt registered [ 5893.383193] Key type .llcrypt registered [ 5893.538911] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 5894.461590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5894.591752] alg: No test for adler32 (adler32-zlib) [ 5895.570660] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 5895.815407] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 5897.528853] Key type lgssc registered [ 5898.698488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5907.165822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5907.210784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5908.761827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5913.882043] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5918.182296] LustreError: 189229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5918.207774] LustreError: 189229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5923.270870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5923.296942] LustreError: 189230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5923.701315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5928.221027] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5937.948989] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5944.744352] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5952.374622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5952.800728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5954.908782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5959.702642] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 5968.310454] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5974.496937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5974.498878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5974.514183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5974.533192] Lustre: Skipped 1 previous similar message [ 5975.265499] Lustre: lustre-OST0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 5975.277843] Lustre: Skipped 1 previous similar message [ 5979.621236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5980.355971] LustreError: 191047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5980.572123] Lustre: server umount lustre-OST0000 complete [ 5990.176651] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90d90350d000.ost_server_uuid 50 [ 5992.687231] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6009.323453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6009.346315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6009.361307] Lustre: Skipped 2 previous similar messages [ 6013.646642] Lustre: server umount lustre-MDT0000 complete [ 6014.433733] LustreError: 191807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.472357] LustreError: 191807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6017.326184] LustreError: 189209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646450 with bad export cookie 5696913131068467727 [ 6017.329415] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6017.336926] LustreError: 189209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6017.805643] Lustre: server umount lustre-MDT0001 complete [ 6027.509946] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:14:18 (1788646458) [ 6035.170230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6035.686580] LustreError: 193145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.747816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.176990] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6045.154372] LustreError: 193146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6045.174287] LustreError: 193146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6053.199245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6053.758190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6059.326041] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6067.957582] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6075.505669] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6084.128874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6084.609929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6091.939367] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6100.218142] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6105.104977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6107.481337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6107.652734] Lustre: lustre-OST0001: new disk, initializing [ 6107.656857] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6107.664558] Lustre: 195986:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6107.782651] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6114.393927] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6116.483833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6116.512441] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6116.674664] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6126.331619] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6129.657561] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6129.664138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6129.677672] Lustre: Skipped 1 previous similar message [ 6129.686626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6135.757745] Lustre: server umount lustre-OST0000 complete [ 6137.324676] LustreError: 194969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.349900] LustreError: 194969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6142.406659] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6142.603295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6147.665352] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6147.895406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6162.912459] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.912482] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.923488] LustreError: Skipped 2 previous similar messages [ 6162.930606] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6162.945334] Lustre: Skipped 2 previous similar messages [ 6162.964668] Lustre: Skipped 5 previous similar messages [ 6167.979630] Lustre: server umount lustre-OST0001 complete [ 6171.618951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6171.638670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6177.919444] Lustre: server umount lustre-MDT0000 complete [ 6178.275023] LustreError: 193145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.312679] LustreError: 193145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 6181.615081] LustreError: 193126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646614 with bad export cookie 5696913131068468798 [ 6181.636082] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6181.890511] Lustre: server umount lustre-MDT0001 complete [ 6204.565411] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:17:15 (1788646635) [ 6211.248570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.497740] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6211.520342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6211.715784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6216.012948] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6225.426079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6225.553882] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6225.675936] Lustre: 199423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6231.394736] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6242.372496] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6249.852305] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6257.588569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6257.710585] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6263.570247] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6264.485173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6271.299090] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6277.605189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6277.680988] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6277.885833] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6277.891751] Lustre: Skipped 2 previous similar messages [ 6283.493091] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6283.950255] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6290.980176] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6294.497670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6294.506332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6294.523546] Lustre: Skipped 2 previous similar messages [ 6294.533861] Lustre: Skipped 4 previous similar messages [ 6300.102944] Lustre: server umount lustre-OST0000 complete [ 6301.900909] LustreError: 201277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6301.936119] LustreError: 201277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6320.104203] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.106220] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6320.126237] Lustre: Skipped 1 previous similar message [ 6322.504809] Lustre: server umount lustre-OST0001 complete [ 6328.289330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6328.302424] LustreError: Skipped 1 previous similar message [ 6328.310700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6328.315580] Lustre: Skipped 5 previous similar messages [ 6333.249761] Lustre: server umount lustre-MDT0000 complete [ 6337.639707] LustreError: 199415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646770 with bad export cookie 5696913131068470205 [ 6337.643140] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6337.653428] LustreError: 199415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6338.213257] Lustre: server umount lustre-MDT0001 complete [ 6366.510496] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:19:57 (1788646797) [ 6384.028576] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 6393.985432] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 6429.121806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.375390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6429.397131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6429.488173] Lustre: lustre-MDT0000: new disk, initializing [ 6429.560848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6429.568754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6433.814624] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6443.598308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6443.723681] Lustre: 207523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6443.797202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6443.799939] Lustre: Skipped 1 previous similar message [ 6443.874633] Lustre: lustre-MDT0001: new disk, initializing [ 6444.070296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6444.084905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6448.172657] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6457.508627] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6464.248521] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6472.100691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6472.501823] Lustre: lustre-OST0000: new disk, initializing [ 6472.512853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6472.521746] Lustre: 209490:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6478.411388] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6481.972394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6481.980824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6482.105957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6487.833526] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6493.237550] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6493.536489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6498.430288] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6498.626049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6502.888663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6502.893297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6502.897838] LustreError: Skipped 1 previous similar message [ 6502.901143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6502.910196] Lustre: Skipped 3 previous similar messages [ 6502.929155] Lustre: Skipped 3 previous similar messages [ 6506.648762] Lustre: server umount lustre-OST0000 complete [ 6524.383131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6524.687854] Lustre: server umount lustre-MDT0000 complete [ 6527.460124] LustreError: 207529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6527.497405] LustreError: 207529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6528.466325] LustreError: 207514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646961 with bad export cookie 5696913131068471990 [ 6528.473496] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6528.484255] LustreError: 207514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6528.927809] Lustre: server umount lustre-MDT0001 complete [ 6537.377310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6542.802059] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6551.573187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6556.157056] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6564.190430] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6570.012411] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6577.151924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6584.838144] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6592.239651] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6596.647484] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6596.872418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6601.119315] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6601.357620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6603.234340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6603.244471] LustreError: Skipped 1 previous similar message [ 6603.250277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6603.265557] Lustre: Skipped 2 previous similar messages [ 6609.311159] Lustre: server umount lustre-OST0000 complete [ 6615.429675] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6619.576538] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6650.203209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6656.134959] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6656.757408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6656.773503] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6663.990064] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6666.793356] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6672.105578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6672.123619] Lustre: Skipped 10 previous similar messages [ 6677.285758] Lustre: server umount lustre-OST0000 complete [ 6686.978720] Lustre: server umount lustre-MDT0000 complete [ 6690.303145] LustreError: 211689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647123 with bad export cookie 5696913131068472872 [ 6690.307386] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6690.310320] LustreError: 211689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6690.607474] Lustre: server umount lustre-MDT0001 complete [ 6698.431252] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:25:30 (1788647130) [ 6704.826040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6705.311171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6705.319688] Lustre: Skipped 6 previous similar messages [ 6709.411681] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6716.817793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6721.508749] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6728.480682] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6733.842653] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6740.370485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6746.827418] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6753.182229] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6757.865985] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6758.073722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6762.251672] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6762.428395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6768.406672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6768.565653] Lustre: lustre-OST0001: new disk, initializing [ 6768.568918] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6768.574293] Lustre: 221040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6775.566123] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6776.967285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6776.985195] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6777.168690] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6786.114888] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6792.635899] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6792.997034] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6799.450472] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6799.738736] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6802.913861] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6802.924935] LustreError: Skipped 1 previous similar message [ 6802.932832] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6802.966416] Lustre: Skipped 5 previous similar messages [ 6816.223149] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6816.426369] Lustre: server umount lustre-OST0001 complete [ 6818.282107] LustreError: 219493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6818.316995] LustreError: 219493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6822.316718] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6822.519326] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6826.436947] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6826.636584] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6856.819937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6862.315387] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6862.331987] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6862.332455] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6862.350554] Lustre: Skipped 1 previous similar message [ 6862.851181] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6870.131650] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6871.785701] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6880.110758] Lustre: server umount lustre-OST0001 complete [ 6902.401819] Lustre: server umount lustre-OST0000 complete [ 6913.794137] Lustre: server umount lustre-MDT0000 complete [ 6917.866661] LustreError: 217654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647350 with bad export cookie 5696913131068473831 [ 6917.875798] LustreError: 217654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6917.885635] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6924.357627] Lustre: server umount lustre-MDT0001 complete [ 6949.862851] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:29:40 (1788647380) [ 6958.250514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6958.640630] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6958.678470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6963.590697] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6972.125315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6972.268837] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6972.306254] Lustre: 226293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6977.372337] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 6984.213425] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6989.699716] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6996.385987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6996.605315] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7003.381713] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7010.930772] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7020.937753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7021.034585] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7028.933411] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7040.645436] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7042.921613] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7047.796330] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7047.995880] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7052.651759] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7052.899103] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7058.539769] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90d903514000.ost_server_uuid 50 [ 7060.150170] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90d903514000.ost_server_uuid in IDLE state after 0 sec [ 7061.186867] Lustre: Permanently deactivating lustre-OST0001 [ 7061.206098] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7064.727711] Lustre: Permanently reactivating lustre-OST0001 [ 7073.250678] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7073.269184] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7073.269807] LustreError: Skipped 4 previous similar messages [ 7073.273331] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7073.273340] Lustre: Skipped 17 previous similar messages [ 7073.324184] Lustre: Skipped 7 previous similar messages [ 7075.079085] Lustre: server umount lustre-OST0001 complete [ 7084.914886] Lustre: server umount lustre-OST0000 complete [ 7094.735426] Lustre: server umount lustre-MDT0000 complete [ 7098.554838] LustreError: 226285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647531 with bad export cookie 5696913131068475119 [ 7098.556937] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7098.560416] LustreError: 226285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7098.950115] Lustre: server umount lustre-MDT0001 complete [ 7125.070788] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:32:36 (1788647556) [ 7132.515572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7132.781547] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7132.804567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7132.811361] Lustre: Skipped 5 previous similar messages [ 7137.708573] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7146.448208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7146.518117] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7146.549185] Lustre: 232983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7151.784782] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7162.013633] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7168.630526] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7175.095374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7175.306361] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7175.344938] Lustre: Permanently deactivating lustre-OST0000 [ 7175.364163] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7181.667990] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7185.440500] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7189.577622] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7197.062604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7197.295362] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7205.887407] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7207.675965] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7214.572071] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7217.016157] Lustre: Permanently reactivating lustre-OST0000 [ 7217.045696] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7217.048761] Lustre: Skipped 2 previous similar messages [ 7224.901808] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7224.903757] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 39 seconds. I think it's dead, and I am evicting it. exp 00000000671dc182, cur 1788647657 deadline 1788647648 last 1788647618 [ 7224.909424] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7224.910317] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7225.034600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7233.702513] Lustre: server umount lustre-OST0001 complete [ 7244.499083] Lustre: server umount lustre-OST0000 complete [ 7254.799758] Lustre: server umount lustre-MDT0000 complete [ 7258.772932] LustreError: 232978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647691 with bad export cookie 5696913131068476764 [ 7258.776882] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7258.783739] LustreError: 232978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7259.393481] Lustre: server umount lustre-MDT0001 complete [ 7271.667727] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 7274.752634] Key type lgssc unregistered [ 7275.073243] LNet: 237863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7275.080835] LNetError: 237863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7276.148780] LNet: Removed LNI 192.168.202.118@tcp [ 7277.158702] Key type .llcrypt unregistered [ 7277.161153] Key type ._llcrypt unregistered [ 7305.880352] Key type ._llcrypt registered [ 7305.902421] Key type .llcrypt registered [ 7306.104750] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7308.228857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7308.581447] alg: No test for adler32 (adler32-zlib) [ 7309.896268] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 7310.220414] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 7311.943172] Key type lgssc registered [ 7313.137332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7323.477333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7323.523776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7325.134405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7329.702658] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7335.409146] LustreError: 239326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7335.443402] LustreError: 239326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7339.134553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7339.319774] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7339.417482] Lustre: 239316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7339.441445] Lustre: Permanently deactivating lustre-MDT0001 [ 7339.499287] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7340.066023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7345.515318] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7349.758440] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7353.683894] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7359.419600] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7367.457488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7367.897516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7370.984488] LustreError: 241167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7370.999658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7376.319914] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7376.360975] LustreError: 241169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7381.475173] LustreError: 241170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.177510] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7384.562340] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7389.668535] LustreError: 241169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7389.706084] LustreError: 241169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7391.561246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7391.876388] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7397.427286] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7397.428023] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7399.514432] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7408.287301] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7411.124679] Lustre: Permanently reactivating lustre-MDT0001 [ 7411.160255] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7411.165684] Lustre: Skipped 2 previous similar messages [ 7416.025389] Lustre: lustre-MDT0001: Client f899e5ac-e41d-4394-8f38-f3dcc260cc2a (at 192.168.202.18@tcp) reconnecting [ 7421.019991] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7421.035787] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7421.048516] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7421.064731] LustreError: 243038:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7421.072799] LustreError: 243038:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7421.073848] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7421.081833] LustreError: 243038:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.118@tcp: cfg command failed: rc = -108 [ 7421.081859] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7421.081859] [ 7421.085371] LustreError: 239318:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7423.045362] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7424.805482] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7432.918707] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7433.275532] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7438.192773] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7438.471552] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7441.637733] Lustre: Permanently deactivating lustre-MDT0001 [ 7441.661864] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7441.667412] Lustre: Skipped 2 previous similar messages [ 7448.778321] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7457.768891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7457.776715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7457.786591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7458.797757] LustreError: 239327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7458.835211] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7458.975310] Lustre: server umount lustre-MDT0000 complete [ 7463.652564] LustreError: 239306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647896 with bad export cookie 13653093981987791698 [ 7463.658815] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7463.667469] LustreError: 239306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7464.442062] Lustre: server umount lustre-MDT0001 complete [ 7471.219641] Lustre: server umount lustre-OST0000 complete [ 7486.386342] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:38:37 (1788647917) [ 7500.728604] Lustre: server umount lustre-OST0001 complete [ 7509.443513] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 7523.570855] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7564.139924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7564.507390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7564.523663] Lustre: Skipped 2 previous similar messages [ 7564.575871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7564.723243] Lustre: lustre-MDT0000: new disk, initializing [ 7564.868391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7564.919350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7570.501446] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7582.323550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7582.515116] Lustre: 247589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7582.557907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7582.561777] Lustre: Skipped 1 previous similar message [ 7582.677743] Lustre: lustre-MDT0001: new disk, initializing [ 7582.855741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7582.921497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7582.936261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7587.957715] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7598.389516] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7604.387673] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7612.043171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7612.301644] Lustre: lustre-OST0000: new disk, initializing [ 7612.304604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7612.315723] Lustre: 249558:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7612.404169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7617.646632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7617.662779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7617.808463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7619.557204] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7630.741658] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7633.400114] LustreError: 249817:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7643.435089] LustreError: 249817:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7643.485503] LustreError: 247836:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7645.811827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7646.050303] Lustre: lustre-OST0001: new disk, initializing [ 7646.054112] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7646.060359] Lustre: 250676:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7646.137631] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7651.505647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7651.518885] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7651.677521] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7652.856237] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7653.576074] LustreError: 247836:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7663.691600] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7665.771627] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7672.290360] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7672.292351] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7672.308180] LustreError: Skipped 1 previous similar message [ 7672.310183] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7672.390708] Lustre: Skipped 3 previous similar messages [ 7674.412332] Lustre: server umount lustre-OST0001 complete [ 7677.933995] LustreError: 250228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7677.971539] LustreError: 250228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 7694.818708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7694.819875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7694.828749] LustreError: Skipped 1 previous similar message [ 7694.831801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7694.831810] Lustre: Skipped 1 previous similar message [ 7694.893987] Lustre: Skipped 1 previous similar message [ 7698.407580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7698.428932] Lustre: Skipped 2 previous similar messages [ 7703.524940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7703.541484] Lustre: Skipped 1 previous similar message [ 7708.127156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7708.415503] Lustre: server umount lustre-OST0000 complete [ 7713.764132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7713.768072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7713.799949] Lustre: Skipped 1 previous similar message [ 7718.881676] LustreError: 247601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7718.897104] LustreError: 247601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 7718.970517] Lustre: server umount lustre-MDT0000 complete [ 7723.184486] LustreError: 247582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648156 with bad export cookie 13653093981987794736 [ 7723.189398] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7723.204227] LustreError: 247582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7730.034409] Lustre: server umount lustre-MDT0001 complete [ 7744.242451] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 7748.237769] Key type lgssc unregistered [ 7748.665977] LNet: 252709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7748.678744] LNetError: 252709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7748.700586] LNet: Removed LNI 192.168.202.118@tcp [ 7750.185750] Key type .llcrypt unregistered [ 7750.187581] Key type ._llcrypt unregistered [ 7804.390632] Key type ._llcrypt registered [ 7804.393803] Key type .llcrypt registered [ 7804.623879] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 7806.848666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7806.925037] alg: No test for adler32 (adler32-zlib) [ 7808.255966] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 7808.749249] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 7810.488062] Key type lgssc registered [ 7812.732666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7822.522401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7822.542979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7823.814601] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7823.836657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7824.192807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7828.729927] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7837.797719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7837.913496] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7837.960575] Lustre: 254916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7838.295471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7843.328428] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7850.300570] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7856.843441] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7864.277482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7864.490624] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7864.718917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7872.322704] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7872.858504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7883.608526] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7939.559583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7939.563031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7939.584902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7942.402990] Lustre: server umount lustre-OST0000 complete [ 7946.255748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7958.496658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7958.808995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7959.856559] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7959.868588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7959.875638] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7959.885311] Lustre: Skipped 1 previous similar message [ 7964.228726] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 7970.720260] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8011.247395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8011.259174] Lustre: Skipped 1 previous similar message [ 8011.259520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8011.277073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8011.277211] LustreError: Skipped 2 previous similar messages [ 8011.290958] Lustre: Skipped 2 previous similar messages [ 8014.885912] Lustre: server umount lustre-OST0000 complete [ 8021.473081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8021.477594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8021.492729] Lustre: Skipped 2 previous similar messages [ 8024.371345] Lustre: server umount lustre-MDT0000 complete [ 8027.252693] LustreError: 254909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648460 with bad export cookie 5462965201111223059 [ 8027.257842] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8027.264876] LustreError: 254909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8027.606091] Lustre: server umount lustre-MDT0001 complete [ 8038.310059] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8042.390207] Key type lgssc unregistered [ 8042.819849] LNet: 260342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8042.824246] LNetError: 260342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8042.839801] LNet: Removed LNI 192.168.202.118@tcp [ 8043.957733] Key type .llcrypt unregistered [ 8043.958993] Key type ._llcrypt unregistered [ 8068.917748] Key type ._llcrypt registered [ 8068.919968] Key type .llcrypt registered [ 8069.000596] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8069.555499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8069.583210] alg: No test for adler32 (adler32-zlib) [ 8070.592696] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 8070.810805] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8072.432043] Key type lgssc registered [ 8073.501689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8080.094577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8080.111648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8081.601765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8085.024803] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8091.108270] LustreError: 261772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8091.120839] LustreError: 261772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8091.355356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8091.636927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8095.473402] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8102.949256] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8109.834392] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8118.423256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8119.150542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8126.275630] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8126.312743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8135.257304] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8167.415146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8167.422532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8167.439834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8167.477849] Lustre: Skipped 1 previous similar message [ 8168.663124] Lustre: server umount lustre-OST0000 complete [ 8177.642146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8177.666590] Lustre: Skipped 2 previous similar messages [ 8177.668171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8177.685397] Lustre: Skipped 1 previous similar message [ 8181.743086] Lustre: server umount lustre-MDT0000 complete [ 8182.781207] LustreError: 264950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8182.817474] LustreError: 264950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8187.889142] LustreError: 261772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8187.920124] LustreError: 261772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8188.940401] LustreError: 261753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648621 with bad export cookie 8055216646822982923 [ 8188.957173] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8188.962694] LustreError: 261753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8189.533411] Lustre: server umount lustre-MDT0001 complete [ 8201.527410] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8204.894704] Key type lgssc unregistered [ 8205.347288] LNet: 266083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8205.364248] LNetError: 266083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8205.385239] LNet: Removed LNI 192.168.202.118@tcp [ 8206.563133] Key type .llcrypt unregistered [ 8206.565845] Key type ._llcrypt unregistered [ 8233.236281] Key type ._llcrypt registered [ 8233.239153] Key type .llcrypt registered [ 8233.387560] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8234.733282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8234.815654] alg: No test for adler32 (adler32-zlib) [ 8236.322208] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 8237.266214] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8239.241073] Key type lgssc registered [ 8242.001744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8251.472376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8251.484574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8253.326655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8257.478223] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8263.653021] LustreError: 267304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8263.670164] LustreError: 267304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8265.813121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8266.144544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8270.510423] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8278.769089] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8286.116405] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8293.523356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8293.827227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8298.940164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8300.921525] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8310.246768] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8324.388467] Lustre: server umount lustre-OST0000 complete [ 8331.747683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8331.757281] Lustre: Skipped 1 previous similar message [ 8331.757960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8331.771197] Lustre: Skipped 1 previous similar message [ 8336.891641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8339.447212] Lustre: server umount lustre-MDT0000 complete [ 8341.986202] LustreError: 268032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8342.012385] LustreError: 268032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8343.009766] Lustre: 266615:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788648759/real 1788648759] req@0000000078e11505 x1875534067824512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788648775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8343.076284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.079731] Lustre: 266615:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8343.122830] LustreError: 267304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8343.147451] LustreError: 267304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8344.483275] LustreError: 267285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648777 with bad export cookie 8723061368035379341 [ 8344.487602] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8344.498230] LustreError: 267285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8345.215598] Lustre: server umount lustre-MDT0001 complete [ 8357.116477] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8359.985654] Key type lgssc unregistered [ 8360.167581] LNet: 270980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8360.170698] LNetError: 270980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8361.190200] LNet: Removed LNI 192.168.202.118@tcp [ 8361.813923] Key type .llcrypt unregistered [ 8361.816683] Key type ._llcrypt unregistered [ 8387.849549] Key type ._llcrypt registered [ 8387.852326] Key type .llcrypt registered [ 8387.983153] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8389.662731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8389.756573] alg: No test for adler32 (adler32-zlib) [ 8390.754953] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 8390.971293] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8392.632841] Key type lgssc registered [ 8393.971301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8402.789627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8402.840283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.351877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8409.210758] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8414.696531] LustreError: 272392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8414.722383] LustreError: 272392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8419.729692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8419.807850] LustreError: 272393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8420.272632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8424.717642] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8433.705148] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8440.648313] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8448.090644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8448.401747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8449.399241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8455.743778] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8463.391043] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8490.468656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8490.482504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8490.518852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8491.496088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8491.496820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8491.514463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8493.932984] Lustre: server umount lustre-OST0000 complete [ 8501.731069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8501.749198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8501.758368] Lustre: Skipped 2 previous similar messages [ 8505.416601] Lustre: server umount lustre-MDT0000 complete [ 8506.849206] LustreError: 273592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8506.866304] LustreError: 273592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8509.372244] LustreError: 272373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648942 with bad export cookie 2740093889816303076 [ 8509.376394] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8509.383473] LustreError: 272373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8509.614365] Lustre: server umount lustre-MDT0001 complete [ 8520.234459] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8522.876430] Key type lgssc unregistered [ 8523.235578] LNet: 276697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8523.260060] LNetError: 276697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8524.326410] LNet: Removed LNI 192.168.202.118@tcp [ 8525.176885] Key type .llcrypt unregistered [ 8525.187118] Key type ._llcrypt unregistered [ 8543.185114] Key type ._llcrypt registered [ 8543.186792] Key type .llcrypt registered [ 8543.302039] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8544.524677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8544.551238] alg: No test for adler32 (adler32-zlib) [ 8545.748503] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 8545.991943] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8547.665663] Key type lgssc registered [ 8548.924551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8556.900221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8556.932343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8558.399869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8562.942371] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8568.806428] LustreError: 277931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8568.840719] LustreError: 277931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8572.666245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8573.246059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8578.415507] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8590.033913] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8597.952494] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8608.679273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8609.506607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8617.238401] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8626.634734] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8628.100458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8648.710727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8648.795231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8648.803899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8648.828477] Lustre: Skipped 1 previous similar message [ 8648.860984] Lustre: Skipped 1 previous similar message [ 8652.975589] Lustre: server umount lustre-OST0000 complete [ 8658.917932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8658.918662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8658.939357] Lustre: Skipped 1 previous similar message [ 8664.037230] LustreError: 277941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8664.078032] LustreError: 277941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8665.390270] Lustre: server umount lustre-MDT0000 complete [ 8668.128665] LustreError: 277918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8668.151280] LustreError: 277918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8669.844229] LustreError: 277898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649102 with bad export cookie 16294771075196772934 [ 8669.852510] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8669.877733] LustreError: 277898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8670.327902] Lustre: server umount lustre-MDT0001 complete [ 8680.920900] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 8684.076375] Key type lgssc unregistered [ 8684.310657] LNet: 281632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8684.315343] LNetError: 281632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8684.331525] LNet: Removed LNI 192.168.202.118@tcp [ 8685.188266] Key type .llcrypt unregistered [ 8685.192943] Key type ._llcrypt unregistered [ 8700.581562] Key type ._llcrypt registered [ 8700.583397] Key type .llcrypt registered [ 8700.795310] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8722.660364] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 8739.886662] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 8741.099219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8741.113940] alg: No test for adler32 (adler32-zlib) [ 8742.212536] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 8742.457268] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 8744.151232] Key type lgssc registered [ 8745.679933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8783.470752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8783.489147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8784.770433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8784.829577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8784.985385] Lustre: lustre-MDT0000: new disk, initializing [ 8785.094841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8785.114450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8789.680887] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8800.172752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8800.356664] Lustre: 285183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8800.442454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8800.446760] Lustre: Skipped 1 previous similar message [ 8800.531822] Lustre: lustre-MDT0001: new disk, initializing [ 8800.645144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8800.695263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8800.709251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8804.637621] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8813.655997] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8819.070321] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8826.865648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8827.191350] Lustre: lustre-OST0000: new disk, initializing [ 8827.195458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8827.203429] Lustre: 287152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8827.336945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8834.252251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8834.294858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8834.600937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8835.839760] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8848.211493] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8856.235279] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8856.453335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8862.092916] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8862.410555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8865.248490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8865.251817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8865.260177] LustreError: Skipped 1 previous similar message [ 8865.260942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8865.304773] Lustre: Skipped 1 previous similar message [ 8870.389373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8870.402047] Lustre: Skipped 2 previous similar messages [ 8871.472781] Lustre: server umount lustre-OST0000 complete [ 8877.543179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8877.553827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.577371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8880.608530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8880.624596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8882.689543] Lustre: server umount lustre-MDT0000 complete [ 8886.860793] LustreError: 285175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649319 with bad export cookie 6415613136713077196 [ 8886.867801] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8886.871718] LustreError: 285175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8887.323315] Lustre: server umount lustre-MDT0001 complete [ 8896.426815] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:02:07 (1788649327) [ 8903.751653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8904.270762] LustreError: 289578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8904.298607] LustreError: 289578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8904.371934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8909.436771] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8909.803224] LustreError: 289579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8914.913140] LustreError: 289578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8918.840220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8919.124435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8923.636218] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8932.083805] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8938.388317] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8945.883910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8946.351733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8953.875135] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 8963.334331] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8975.328792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8975.335125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8975.346757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8975.352156] Lustre: Skipped 2 previous similar messages [ 8979.129961] Lustre: server umount lustre-OST0000 complete [ 8985.569144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8985.570502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8985.576590] Lustre: Skipped 1 previous similar message [ 8985.580014] LustreError: Skipped 1 previous similar message [ 8985.581397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8985.589243] Lustre: Skipped 1 previous similar message [ 8988.979316] Lustre: server umount lustre-MDT0000 complete [ 8990.691870] LustreError: 290306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8990.728427] LustreError: 290306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8992.782035] LustreError: 289560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649425 with bad export cookie 6415613136713078078 [ 8992.785144] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8992.792226] LustreError: 289560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8993.048660] Lustre: server umount lustre-MDT0001 complete [ 9004.077805] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 9006.864520] Key type lgssc unregistered [ 9007.169107] LNet: 293115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9007.176658] LNetError: 293115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9007.195415] LNet: Removed LNI 192.168.202.118@tcp [ 9008.013699] Key type .llcrypt unregistered [ 9008.019599] Key type ._llcrypt unregistered [ 9032.301520] Key type ._llcrypt registered [ 9032.308641] Key type .llcrypt registered [ 9032.391572] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 9043.966210] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9044.903819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9044.939218] alg: No test for adler32 (adler32-zlib) [ 9045.939880] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 9046.162315] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 9047.839162] Key type lgssc registered [ 9048.794612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9086.505076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9086.522732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9087.692630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9087.713511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9087.786610] Lustre: lustre-MDT0000: new disk, initializing [ 9087.832583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9087.847477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9092.098265] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9101.053724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9101.172405] Lustre: 296915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9101.225635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9101.230096] Lustre: Skipped 1 previous similar message [ 9101.283315] Lustre: lustre-MDT0001: new disk, initializing [ 9101.355236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9101.388553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9101.401043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9105.216684] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9114.508208] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9119.576365] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9125.670700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9126.009105] Lustre: lustre-OST2710: new disk, initializing [ 9126.027764] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9126.035674] Lustre: 298885:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9126.188867] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9133.158445] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9133.623774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9133.644261] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9133.788565] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9142.745116] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9150.326418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9150.552740] Lustre: lustre-OST03e8: new disk, initializing [ 9150.555433] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9150.563449] Lustre: 299950:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9150.677432] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9156.956627] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9157.278196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9157.296218] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9157.487238] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9164.258248] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9170.796302] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9171.032948] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9176.321971] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9176.589240] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9181.530839] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9181.687426] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9188.022417] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9188.360484] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9193.440279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9193.441895] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.457793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9193.477887] Lustre: Skipped 3 previous similar messages [ 9195.656267] Lustre: server umount lustre-MDT0000 complete [ 9200.465597] LustreError: 297711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649633 with bad export cookie 3942385602140450655 [ 9200.485525] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9200.500147] LustreError: 297711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9201.207562] Lustre: server umount lustre-MDT0001 complete [ 9216.980684] Lustre: server umount lustre-OST2710 complete [ 9220.069377] Lustre: 294714:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788649636/real 1788649636] req@000000001cc7005c x1875534916161152/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788649652 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9220.105151] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9221.151754] Lustre: 294712:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788649638/real 1788649638] req@0000000051c6e056 x1875534916161408/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788649654 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9221.883431] Lustre: server umount lustre-OST03e8 complete [ 9248.389718] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [ 9263.482538] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9303.072612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9303.547458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9303.571163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9303.665339] Lustre: lustre-MDT0000: new disk, initializing [ 9303.749909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9303.767432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9309.803062] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9321.977559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9322.035602] Lustre: 305201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9322.078951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9322.089098] Lustre: Skipped 1 previous similar message [ 9322.137826] Lustre: lustre-MDT0001: new disk, initializing [ 9322.196684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9322.223032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9322.227942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9326.471660] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9339.125685] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9345.062609] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9353.771937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9354.513220] Lustre: lustre-OST0000: new disk, initializing [ 9354.525579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9354.575416] Lustre: 307171:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9354.877827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9363.770657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9363.785909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9363.973207] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9364.460335] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9379.651872] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9385.177697] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9385.654564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9391.219327] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9391.465775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9394.660326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9394.663301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9394.672808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9394.672817] Lustre: Skipped 3 previous similar messages [ 9394.681350] LustreError: Skipped 1 previous similar message [ 9394.715450] Lustre: Skipped 1 previous similar message [ 9399.781741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9399.787242] Lustre: Skipped 1 previous similar message [ 9400.071957] Lustre: server umount lustre-OST0000 complete [ 9404.386322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9404.388830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9404.399226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9404.403327] Lustre: Skipped 1 previous similar message [ 9410.023023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9410.033596] Lustre: Skipped 2 previous similar messages [ 9410.709798] Lustre: server umount lustre-MDT0000 complete [ 9415.142302] LustreError: 305213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9415.167110] LustreError: 305213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9416.043345] LustreError: 305193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649848 with bad export cookie 3942385602140452335 [ 9416.062862] LustreError: 305193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9416.068891] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9416.829743] Lustre: server umount lustre-MDT0001 complete [ 9429.772058] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:11:00 (1788649860) [ 9431.659776] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9435.060726] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:11:05 (1788649865) [ 9440.419812] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9477.088778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9477.234302] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9477.246772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9477.337983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9482.198864] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9490.691850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9490.782811] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9490.807131] Lustre: 311033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9495.508631] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9517.292794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9517.589637] Lustre: Denying initial registration attempt from nid 192.168.202.118@tcp, specified as failover [ 9517.602670] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9517.611690] LustreError: 312484:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9517.616838] LustreError: 312484:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9517.625406] LustreError: 312484:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9517.755665] Lustre: server umount lustre-OST0000 complete [ 9517.763365] LustreError: 312484:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9526.752490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9526.769556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9526.780194] Lustre: Skipped 1 previous similar message [ 9526.784731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9531.721479] Lustre: server umount lustre-MDT0000 complete [ 9532.900370] LustreError: 311044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9532.927151] LustreError: 311044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9535.696741] LustreError: 311023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649968 with bad export cookie 3942385602140453217 [ 9535.722758] LustreError: 311023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9535.729566] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9536.168262] Lustre: server umount lustre-MDT0001 complete [ 9547.201717] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 9551.055872] Key type lgssc unregistered [ 9551.587683] LNet: 313605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9551.598056] LNetError: 313605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9551.617261] LNet: Removed LNI 192.168.202.118@tcp [ 9552.623500] Key type .llcrypt unregistered [ 9552.626697] Key type ._llcrypt unregistered [ 9566.105506] Key type ._llcrypt registered [ 9566.108137] Key type .llcrypt registered [ 9566.216750] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9567.713277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9567.775172] alg: No test for adler32 (adler32-zlib) [ 9569.318090] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 9569.754918] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 9571.487290] Key type lgssc registered [ 9573.354654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9621.530808] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9630.797386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9630.839313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9632.065242] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9632.105819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9632.272108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9637.070722] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9646.444551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9646.556364] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9646.582544] Lustre: 316368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9646.767604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9651.753365] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9660.082299] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9666.670748] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9687.361278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9687.677565] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9687.980821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9695.539092] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9704.725639] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9710.563422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9710.569388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9710.575475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9710.591431] LustreError: Skipped 1 previous similar message [ 9710.634266] Lustre: Skipped 1 previous similar message [ 9713.726917] Lustre: server umount lustre-OST0000 complete [ 9718.752969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9718.759693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9718.781189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9718.789747] Lustre: Skipped 1 previous similar message [ 9720.802313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9720.802807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9724.485966] Lustre: server umount lustre-MDT0000 complete [ 9729.003199] LustreError: 318986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650161 with bad export cookie 17006596956496056166 [ 9729.005656] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9729.027509] LustreError: 318986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9729.417328] Lustre: server umount lustre-MDT0001 complete [ 9741.193167] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 9744.348374] Key type lgssc unregistered [ 9744.666500] LNet: 319939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9744.673551] LNetError: 319939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9745.701846] LNet: Removed LNI 192.168.202.118@tcp [ 9746.885582] Key type .llcrypt unregistered [ 9746.891594] Key type ._llcrypt unregistered [ 9774.269447] Key type ._llcrypt registered [ 9774.272276] Key type .llcrypt registered [ 9774.403563] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [ 9775.448627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9775.546981] alg: No test for adler32 (adler32-zlib) [ 9776.729486] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [ 9776.987175] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [ 9778.631252] Key type lgssc registered [ 9779.774790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9787.659678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9787.693280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9789.259481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9793.943351] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9799.649479] LustreError: 321356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9799.683145] LustreError: 321356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9802.557394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9803.016453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9807.835380] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9815.459491] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9822.229796] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9831.270152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9831.823643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9839.345437] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9848.004716] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9857.505557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9857.511475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9857.531893] Lustre: Skipped 2 previous similar messages [ 9857.551228] Lustre: Skipped 2 previous similar messages [ 9861.352272] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [ 9862.630945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9862.639829] Lustre: Skipped 1 previous similar message [ 9863.487498] Lustre: server umount lustre-MDT0000 complete [ 9866.435305] LustreError: 321351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9867.530030] LustreError: 323936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650300 with bad export cookie 12063154106622191709 [ 9867.550581] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9867.554975] LustreError: 323936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9867.745348] LustreError: 321352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9867.745724] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9867.784263] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9871.564671] LustreError: 322597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9872.865307] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9872.876728] Lustre: Skipped 1 previous similar message [ 9873.923267] Lustre: server umount lustre-MDT0001 complete [ 9879.992322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9888.037792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9892.320614] LustreError: 320663:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000016a2bd8a x1875535681584896/t0(0) o250->MGC192.168.202.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9892.788028] LustreError: 323189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9892.809508] LustreError: 323189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9892.988425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9893.077644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9898.722431] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9902.179599] LustreError: 323189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9902.185381] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9902.202901] LustreError: 323189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9907.201542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9907.616746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9911.972728] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [ 9912.836823] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9919.153451] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9920.433953] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9926.168890] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9927.793779] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9933.290762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9933.295859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9933.299862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9933.306457] Lustre: Skipped 1 previous similar message [ 9933.335461] Lustre: Skipped 1 previous similar message [ 9938.252364] Lustre: server umount lustre-OST0000 complete [ 9943.523075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9943.531255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9953.766224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9953.777778] Lustre: Skipped 5 previous similar messages [ 9956.319187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9956.612068] Lustre: server umount lustre-MDT0000 complete [ 9958.895108] LustreError: 324952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9958.909412] LustreError: 324952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9960.587389] LustreError: 321337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650393 with bad export cookie 12063154106622198856 [ 9960.597286] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9960.606704] LustreError: 321337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9960.896245] Lustre: server umount lustre-MDT0001 complete [ 9972.238356] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [ 9975.069924] Key type lgssc unregistered [ 9975.389778] LNet: 327624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9975.401724] LNetError: 327624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9975.429371] LNet: Removed LNI 192.168.202.118@tcp [ 9976.402581] Key type .llcrypt unregistered [ 9976.412052] Key type ._llcrypt unregistered [10000.541512] Key type ._llcrypt registered [10000.543984] Key type .llcrypt registered [10000.647611] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [10001.541310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10001.596466] alg: No test for adler32 (adler32-zlib) [10002.697793] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [10002.919371] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [10004.639412] Key type lgssc registered [10005.948780] Lustre: Echo OBD driver; http://www.lustre.org/ [10013.343826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10013.360466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10014.793104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10019.625682] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10024.929370] LustreError: 329035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10024.965886] LustreError: 329035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10028.123199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.587196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10033.468795] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10042.041346] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10047.949248] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10053.864222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10053.881046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10053.888530] Lustre: Skipped 1 previous similar message [10059.232753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10059.236146] Lustre: Skipped 1 previous similar message [10064.353300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10064.357063] Lustre: Skipped 1 previous similar message [10065.375179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10065.613452] Lustre: server umount lustre-MDT0000 complete [10069.473493] LustreError: 329035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10069.490968] LustreError: 329035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10069.687243] LustreError: 329016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650502 with bad export cookie 16189197044592496770 [10069.703455] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10069.706209] LustreError: 329016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10070.076844] Lustre: server umount lustre-MDT0001 complete [10083.923794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.316963] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10084.355520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10084.580538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10089.162087] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10096.908905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10096.978380] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10097.003283] Lustre: 331623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10097.243103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10101.486502] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10108.133734] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10113.487645] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10124.158375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10124.335855] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10124.342802] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10124.353257] LustreError: 333511:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10124.361688] LustreError: 333511:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10124.367587] LustreError: 333511:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10124.439348] Lustre: server umount lustre-OST0000 complete [10124.443606] LustreError: 333511:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10134.636947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10134.898931] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10135.120697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10141.361737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10142.374244] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10150.576713] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10158.383275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10158.488806] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10158.551270] Lustre: lustre-OST0001: new disk, initializing [10158.556488] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10158.563982] Lustre: 335051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10158.635460] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10164.318835] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10166.333768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10166.353432] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10166.433984] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10172.388349] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10176.997242] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10177.005111] Lustre: Skipped 1 previous similar message [10177.012461] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10177.013851] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10177.024377] LustreError: Skipped 1 previous similar message [10177.042203] Lustre: Skipped 1 previous similar message [10182.128650] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10182.133719] Lustre: Skipped 1 previous similar message [10190.303552] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10190.515917] Lustre: server umount lustre-OST0001 complete [10192.358691] LustreError: 335527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10192.386353] LustreError: 335527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10197.472949] LustreError: 333976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10197.489318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10197.499371] LustreError: 333976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10197.501908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10197.501917] Lustre: Skipped 2 previous similar messages [10197.502814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10197.576810] Lustre: Skipped 2 previous similar messages [10200.419309] Lustre: server umount lustre-OST0000 complete [10204.643276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10204.648063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10210.188680] Lustre: server umount lustre-MDT0000 complete [10212.838597] LustreError: 331631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10212.848419] LustreError: 331631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10213.824869] LustreError: 331615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650646 with bad export cookie 16189197044592497323 [10213.834587] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10213.843873] LustreError: 331615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10214.290778] Lustre: server umount lustre-MDT0001 complete [10224.724507] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [10227.460809] Key type lgssc unregistered [10227.830252] LNet: 337009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10227.845256] LNetError: 337009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10227.866308] LNet: Removed LNI 192.168.202.118@tcp [10228.780537] Key type .llcrypt unregistered [10228.782932] Key type ._llcrypt unregistered [10298.985191] Key type ._llcrypt registered [10298.991093] Key type .llcrypt registered [10299.083832] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [10314.549627] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [10316.059873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10316.129551] alg: No test for adler32 (adler32-zlib) [10317.158440] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [10317.424205] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [10319.111205] Key type lgssc registered [10320.571817] Lustre: Echo OBD driver; http://www.lustre.org/ [10354.602894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10354.692853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10355.993470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10356.018189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10356.147762] Lustre: lustre-MDT0000: new disk, initializing [10356.254837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10356.290242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10361.453403] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10371.784844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10371.895612] Lustre: 341940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10371.955880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10371.959362] Lustre: Skipped 1 previous similar message [10372.053495] Lustre: lustre-MDT0001: new disk, initializing [10372.171259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10372.204897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10372.227347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10376.509352] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10385.975592] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10391.559706] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10399.328908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10399.715439] Lustre: lustre-OST0000: new disk, initializing [10399.722565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10399.732365] Lustre: 343905:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10399.821697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10407.136193] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10409.114829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10409.138946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10409.320400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10418.999216] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10424.879994] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10425.302768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10430.212543] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10430.509752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10435.047031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10435.047187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10435.065845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10435.066276] LustreError: Skipped 1 previous similar message [10435.086618] Lustre: Skipped 1 previous similar message [10438.967734] Lustre: server umount lustre-OST0000 complete [10443.745541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10443.762745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10443.790680] Lustre: Skipped 1 previous similar message [10443.803789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10445.285220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10445.305403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10449.700426] Lustre: server umount lustre-MDT0000 complete [10453.936443] LustreError: 341933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650886 with bad export cookie 18018038366367359074 [10453.937636] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10453.959464] LustreError: 341933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10454.514264] Lustre: server umount lustre-MDT0001 complete [10464.632611] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:28:15 (1788650895) [10473.757594] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:28:24 (1788650904) [10484.989341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10485.634606] LustreError: 346663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10485.647959] LustreError: 346663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10485.733765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10490.852153] LustreError: 346664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10491.505669] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10495.970386] LustreError: 346663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10502.124184] LustreError: 346664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10502.157404] LustreError: 346664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10503.249929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10503.500311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10510.873261] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10522.529240] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10529.474855] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10536.490301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10536.854158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10543.046154] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10550.267231] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10563.850864] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10565.781753] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10567.405802] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10569.258923] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10570.871823] Lustre: Failing over lustre-MDT0000 [10571.752099] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10571.756661] LustreError: 347579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10571.766867] Lustre: Skipped 1 previous similar message [10571.792860] LustreError: 347579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10573.215159] Lustre: server umount lustre-MDT0000 complete [10581.988092] LustreError: 346664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10582.012019] LustreError: 346664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10587.049595] Lustre: 340161:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788651004/real 1788651004] req@000000005b8e1b23 x1875536248906496/t0(0) o400->MGC192.168.202.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788651020 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10587.096940] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10592.437125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10597.367648] LustreError: 340160:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000a43de48 x1875536248911872/t0(0) o250->MGC192.168.202.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10597.656823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10597.693452] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10599.118531] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10603.012444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10603.058456] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10603.141714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10604.209166] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10615.188414] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10616.900657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10620.820886] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10622.784047] Lustre: Failing over lustre-MDT0000 [10622.998637] Lustre: server umount lustre-MDT0000 complete [10623.467193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10623.474057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10623.482313] Lustre: Skipped 1 previous similar message [10623.490187] LustreError: 346660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10623.511884] LustreError: 346660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [10639.842395] Lustre: 340161:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788651056/real 1788651056] req@000000006ded4c54 x1875536248926976/t0(0) o400->MGC192.168.202.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788651072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10639.879812] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10642.185772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10649.075047] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfa0cee6e38afaaf7 [10649.091758] Lustre: MGC192.168.202.118@tcp: Connection restored to 0@lo (at 0@lo) [10649.110087] Lustre: Skipped 2 previous similar messages [10649.489730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10649.588760] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10650.317504] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10654.718026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10654.768131] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10654.845384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10654.924333] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10664.843442] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10666.607341] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10675.171890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10675.183129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10675.193567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10675.202573] Lustre: Skipped 4 previous similar messages [10675.213374] Lustre: Skipped 3 previous similar messages [10679.534889] Lustre: server umount lustre-MDT0000 complete [10680.290216] LustreError: 347579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10680.313226] LustreError: 347579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [10683.257629] LustreError: 346643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651116 with bad export cookie 18018038366367361783 [10683.261216] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10683.283587] LustreError: 346643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10683.568213] Lustre: server umount lustre-MDT0001 complete [10697.475577] Lustre: server umount lustre-OST0000 complete [10714.129878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10714.703854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10720.045749] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10729.483589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10734.322075] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10742.293560] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10749.302761] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10757.744722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10759.236712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10765.570717] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10774.050680] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10782.229747] Lustre: server umount lustre-OST0000 complete [10789.864112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10789.872505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10789.895424] Lustre: Skipped 1 previous similar message [10789.917796] Lustre: Skipped 1 previous similar message [10796.330515] Lustre: server umount lustre-MDT0000 complete [10800.097347] LustreError: 353028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10800.118672] LustreError: 353028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10800.269036] LustreError: 353010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651233 with bad export cookie 18018038366367362532 [10800.278448] LustreError: 353010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10800.290160] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10811.163623] Lustre: server umount lustre-MDT0001 complete [10825.419282] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [10828.882279] Key type lgssc unregistered [10829.279617] LNet: 356561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10829.291968] LNetError: 356561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10829.305953] LNet: Removed LNI 192.168.202.118@tcp [10830.219473] Key type .llcrypt unregistered [10830.223933] Key type ._llcrypt unregistered [10857.059323] Key type ._llcrypt registered [10857.061766] Key type .llcrypt registered [10857.146271] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [10858.143967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10858.232495] alg: No test for adler32 (adler32-zlib) [10859.335114] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [10859.543420] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [10861.199240] Key type lgssc registered [10862.343436] Lustre: Echo OBD driver; http://www.lustre.org/ [10871.120773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10871.156939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10872.979461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10878.691875] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10882.530683] LustreError: 358074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10882.557530] LustreError: 358074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10887.662745] LustreError: 358075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10888.846064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10889.079666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10893.952214] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10902.647750] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10908.866467] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10916.315385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10916.867513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10917.892473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10925.142087] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10933.561924] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10945.506936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10945.519598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10945.534227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10947.020608] Lustre: server umount lustre-MDT0000 complete [10948.588737] LustreError: 358074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10948.606378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10951.569644] LustreError: 358056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651384 with bad export cookie 9767838588900682979 [10951.574686] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10951.580108] LustreError: 358056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10951.934173] Lustre: server umount lustre-MDT0001 complete [10966.489861] Lustre: server umount lustre-OST0000 complete [10980.778783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10981.189422] LustreError: 361785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10981.218606] LustreError: 361785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10981.272465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10986.465614] LustreError: 361786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10986.588932] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [10996.455735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10996.705469] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10996.968243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11002.782723] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11012.139237] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11018.509126] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11026.175125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11026.602180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11027.649419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11033.465587] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11041.015240] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11053.029142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11053.037901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11053.047888] Lustre: Skipped 1 previous similar message [11053.059803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11057.635705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11057.645154] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11059.222802] Lustre: server umount lustre-MDT0000 complete [11062.733504] LustreError: 361765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651495 with bad export cookie 9767838588900684092 [11062.737969] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11062.746037] LustreError: 361765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11062.773668] LustreError: 361780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11062.777347] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11062.784233] LustreError: 361780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11062.813547] Lustre: Skipped 1 previous similar message [11062.824629] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11062.831451] Lustre: Skipped 2 previous similar messages [11063.030249] Lustre: server umount lustre-MDT0001 complete [11069.298783] Lustre: server umount lustre-OST0000 complete [11087.994379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11088.656590] LustreError: 365698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11088.753201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11092.888777] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11100.533927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11105.158768] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11111.848434] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11117.752962] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11125.549900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11125.740462] LustreError: 367515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11125.759621] LustreError: 367515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11126.023302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11126.035346] Lustre: Skipped 1 previous similar message [11132.618428] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11133.440501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11139.957732] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11146.580963] Lustre: server umount lustre-OST0000 complete [11152.353598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11152.360534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11152.382182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11161.104880] Lustre: server umount lustre-MDT0000 complete [11162.080318] Lustre: 357384:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788651579/real 1788651579] req@0000000052249e63 x1875536817251584/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788651595 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11164.128520] Lustre: 357385:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788651581/real 1788651581] req@0000000079878743 x1875536817252608/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788651597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11164.576581] LustreError: 365677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651597 with bad export cookie 9767838588900685114 [11164.580512] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11164.586861] LustreError: 365677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11164.937744] Lustre: server umount lustre-MDT0001 complete [11176.564619] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [11179.469779] Key type lgssc unregistered [11179.771379] LNet: 369228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11179.778450] LNetError: 369228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11179.792823] LNet: Removed LNI 192.168.202.118@tcp [11180.714711] Key type .llcrypt unregistered [11180.716913] Key type ._llcrypt unregistered [11205.683535] Key type ._llcrypt registered [11205.685672] Key type .llcrypt registered [11205.857963] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [11206.927591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11207.026621] alg: No test for adler32 (adler32-zlib) [11208.017243] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [11208.345855] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [11209.999246] Key type lgssc registered [11211.278740] Lustre: Echo OBD driver; http://www.lustre.org/ [11219.191150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11219.201652] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11219.206879] LustreError: 370647:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11219.217945] LustreError: 370647:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11219.222817] LustreError: 370647:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11219.227499] LustreError: 370647:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11219.235167] LustreError: 370647:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11226.284850] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11226.299330] LustreError: 371044:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11226.350529] LustreError: 371044:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11226.366236] LustreError: 371044:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11226.382789] LustreError: 371044:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11226.400278] LustreError: 371044:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11246.392902] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [11249.320594] Key type lgssc unregistered [11249.675823] LNet: 372046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11249.682664] LNetError: 372046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11249.714828] LNet: Removed LNI 192.168.202.118@tcp [11250.495586] Key type .llcrypt unregistered [11250.497063] Key type ._llcrypt unregistered [11270.096527] Key type ._llcrypt registered [11270.098239] Key type .llcrypt registered [11270.191973] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [11285.751772] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [11286.595514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11286.674104] alg: No test for adler32 (adler32-zlib) [11287.757726] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [11288.064328] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [11289.727208] Key type lgssc registered [11290.875604] Lustre: Echo OBD driver; http://www.lustre.org/ [11324.930944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11324.976757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11326.408303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11326.460379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11326.519623] Lustre: lustre-MDT0000: new disk, initializing [11326.587973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11326.607067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11331.394568] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11341.885344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11342.017126] Lustre: 375223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11342.079892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11342.083467] Lustre: Skipped 1 previous similar message [11342.186295] Lustre: lustre-MDT0001: new disk, initializing [11342.310189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11342.376020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11342.395740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11346.963787] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11359.942216] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11367.688511] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11376.773952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11377.388195] Lustre: lustre-OST0000: new disk, initializing [11377.396682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11377.413336] Lustre: 377195:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11377.587329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11384.536962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11384.569435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11384.778435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11385.718399] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11397.362412] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11402.666380] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11402.977421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11408.382546] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11408.612738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11410.412576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11410.424064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11410.435191] LustreError: Skipped 1 previous similar message [11410.437555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11410.481022] Lustre: Skipped 1 previous similar message [11415.521514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11415.531419] Lustre: Skipped 2 previous similar messages [11416.706452] Lustre: server umount lustre-OST0000 complete [11420.642162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11420.654617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11420.672792] Lustre: Skipped 1 previous similar message [11420.686652] Lustre: Skipped 1 previous similar message [11425.760224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11425.766457] Lustre: Skipped 1 previous similar message [11426.873703] Lustre: server umount lustre-MDT0000 complete [11432.128735] LustreError: 377193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651865 with bad export cookie 7382167766361645550 [11432.136208] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11432.504994] Lustre: server umount lustre-MDT0001 complete [11442.906352] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11444.339249] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:44:35 (1788651875) [11451.170225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11451.678782] LustreError: 379658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11451.703211] LustreError: 379658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11451.802281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11456.901114] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11456.994817] LustreError: 379659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11461.092407] LustreError: 379658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11466.209521] LustreError: 379659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11468.051565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11468.487465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11473.630177] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11481.597829] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11487.752230] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11495.176238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11495.776964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11502.619838] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11510.569592] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11519.115044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11519.372461] Lustre: lustre-OST0001: new disk, initializing [11519.381320] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11519.394829] Lustre: 382495:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11519.566135] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11526.778976] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11529.345407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11529.359812] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11529.423608] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11537.787492] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11544.812972] Lustre: lustre-OST0001: Not available for connect from 192.168.202.18@tcp (stopping) [11545.063320] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11545.085354] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11548.221421] Lustre: server umount lustre-OST0001 complete [11549.896831] LustreError: 381480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11560.419894] LustreError: 382220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11560.428874] LustreError: 382220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11565.553290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11565.571018] Lustre: Skipped 1 previous similar message [11565.577456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11565.580329] Lustre: Skipped 2 previous similar messages [11579.360204] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11579.728685] Lustre: server umount lustre-OST0000 complete [11586.016872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11586.020303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11586.025938] Lustre: Skipped 2 previous similar messages [11586.029200] Lustre: Skipped 4 previous similar messages [11598.303477] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11598.833530] Lustre: server umount lustre-MDT0000 complete [11601.378900] LustreError: 379658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11601.398882] LustreError: 379658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11602.663181] LustreError: 379639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652035 with bad export cookie 7382167766361646432 [11602.663790] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11602.680147] LustreError: 379639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11609.381911] Lustre: server umount lustre-MDT0001 complete [11620.622418] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [11623.177537] Key type lgssc unregistered [11623.472328] LNet: 384482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11623.478417] LNetError: 384482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11623.494266] LNet: Removed LNI 192.168.202.118@tcp [11624.479063] Key type .llcrypt unregistered [11624.480371] Key type ._llcrypt unregistered [11674.224957] Key type ._llcrypt registered [11674.227077] Key type .llcrypt registered [11674.394109] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [11676.209112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11676.318742] alg: No test for adler32 (adler32-zlib) [11677.562156] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [11677.944524] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [11679.711332] Key type lgssc registered [11680.969630] Lustre: Echo OBD driver; http://www.lustre.org/ [11688.598087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11688.632072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11690.021380] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11690.044988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11690.228655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11695.384674] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11704.412239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11704.488532] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11704.517774] Lustre: 387281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11704.743384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11710.083343] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11717.949612] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11724.268054] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11726.817044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11726.841456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11726.849678] Lustre: Skipped 1 previous similar message [11731.936783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11731.951136] Lustre: Skipped 1 previous similar message [11732.672985] Lustre: server umount lustre-MDT0000 complete [11736.252857] LustreError: 387272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652169 with bad export cookie 4924790383339621567 [11736.261164] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11736.268935] LustreError: 387272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11736.538740] Lustre: server umount lustre-MDT0001 complete [11747.112479] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:49:38 (1788652178) [11754.093486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11754.801691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11759.819539] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11760.109117] LustreError: 389794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.142365] LustreError: 389794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11765.228551] LustreError: 389795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.955462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.415062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11772.594305] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11781.349328] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11786.780538] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11794.741451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11794.918638] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11795.096573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11802.251128] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11810.896530] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11819.611513] Lustre: Permanently deactivating lustre-OST0000 [11819.627553] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11829.280919] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11832.108472] LustreError: 392829:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11832.114683] LustreError: 392829:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11832.118883] LustreError: 392829:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11835.323855] LustreError: 392877:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11835.333430] LustreError: 392877:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11843.665632] Lustre: server umount lustre-OST0000 complete [11846.559105] LustreError: 393127:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11846.564389] LustreError: 393127:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11850.208804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11850.216652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11850.231869] Lustre: Skipped 1 previous similar message [11850.237972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11851.744904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11856.865082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11856.867266] Lustre: Skipped 2 previous similar messages [11859.057118] Lustre: server umount lustre-MDT0000 complete [11861.993926] LustreError: 390520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11862.018602] LustreError: 390520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11863.619451] LustreError: 389774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652296 with bad export cookie 4924790383339622148 [11863.627481] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11863.647132] LustreError: 389774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11873.573639] Lustre: server umount lustre-MDT0001 complete [11883.590828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11889.214419] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11892.384569] LustreError: 394287:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11898.983325] Lustre: 394626:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.118@tcp [11901.132322] Lustre: server umount MGS complete [11912.386890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11912.881445] LustreError: 395350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11912.907851] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11912.922153] Lustre: Skipped 1 previous similar message [11912.980757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11918.308619] LustreError: 395351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11918.359296] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11927.773350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11928.206555] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11934.142717] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [11942.704615] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11949.123165] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11951.448992] Lustre: Permanently reactivating lustre-OST0000 [11951.488940] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11951.498040] Lustre: Skipped 2 previous similar messages [11980.769306] LustreError: 397139:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11988.837287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11989.164146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11989.181029] Lustre: Skipped 1 previous similar message [11997.117169] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12006.151295] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12013.953767] Lustre: server umount lustre-OST0000 complete [12020.709034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12020.717891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12020.753612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12028.472530] Lustre: server umount lustre-MDT0000 complete [12031.457689] LustreError: 395351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12031.476892] LustreError: 395351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12032.017242] LustreError: 395331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652464 with bad export cookie 4924790383339623198 [12032.021424] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12032.025640] LustreError: 395331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12041.325587] Lustre: server umount lustre-MDT0001 complete [12050.899798] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12053.790673] Key type lgssc unregistered [12054.088923] LNet: 399718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12054.103678] LNetError: 399718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12055.144833] LNet: Removed LNI 192.168.202.118@tcp [12056.221704] Key type .llcrypt unregistered [12056.225075] Key type ._llcrypt unregistered [12073.287359] Key type ._llcrypt registered [12073.289479] Key type .llcrypt registered [12073.424360] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [12089.540391] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12090.757640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12090.882964] alg: No test for adler32 (adler32-zlib) [12092.042400] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [12092.320920] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12094.039519] Key type lgssc registered [12095.040192] Lustre: Echo OBD driver; http://www.lustre.org/ [12129.085783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12129.141561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12130.389154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12130.410788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12130.510162] Lustre: lustre-MDT0000: new disk, initializing [12130.585233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12130.601090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12135.629606] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12146.546191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12146.674963] Lustre: 402872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12146.755783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12146.759487] Lustre: Skipped 1 previous similar message [12146.841504] Lustre: lustre-MDT0001: new disk, initializing [12146.932028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12146.969525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12146.982363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12151.441799] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12161.124559] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12166.853730] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12173.718211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12174.168269] Lustre: lustre-OST0000: new disk, initializing [12174.174469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12174.182803] Lustre: 404838:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12174.294931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12180.644517] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12183.694184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12183.708705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12183.832314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12189.642270] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12194.861162] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12195.056597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12201.714101] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12201.956129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12209.633882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12209.635091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12209.636835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12209.638236] LustreError: Skipped 1 previous similar message [12209.666836] Lustre: Skipped 1 previous similar message [12211.022931] Lustre: server umount lustre-OST0000 complete [12218.859757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12218.866088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12218.895960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12218.906224] Lustre: Skipped 1 previous similar message [12219.874300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12221.729196] Lustre: server umount lustre-MDT0000 complete [12225.326623] LustreError: 402862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652658 with bad export cookie 10956845803732553202 [12225.338417] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12225.345424] LustreError: 402862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12225.604256] Lustre: server umount lustre-MDT0001 complete [12233.555232] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12234.997944] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:57:46 (1788652666) [12242.639530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12243.041687] LustreError: 407299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12243.070632] LustreError: 407299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12243.136627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12248.153989] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12248.555920] LustreError: 407300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12253.671958] LustreError: 407299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12258.247340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12258.547815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12262.907604] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12270.276136] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12276.604600] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12284.209125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12284.715110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12292.809544] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12300.789574] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12302.628993] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12305.526450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12313.571497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12313.574250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12313.634924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12313.667947] Lustre: Skipped 3 previous similar messages [12317.633663] Lustre: server umount lustre-OST0000 complete [12323.811105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12323.866945] Lustre: Skipped 1 previous similar message [12323.907721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12323.929955] Lustre: Skipped 1 previous similar message [12328.932029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12328.941811] Lustre: Skipped 1 previous similar message [12331.240547] Lustre: server umount lustre-MDT0000 complete [12334.063452] LustreError: 407296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12334.105406] LustreError: 407296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12334.844906] LustreError: 408034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652767 with bad export cookie 10956845803732554091 [12334.848297] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12334.852849] LustreError: 408034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12335.196213] Lustre: server umount lustre-MDT0001 complete [12348.294278] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12351.350687] Key type lgssc unregistered [12351.722967] LNet: 411028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12351.730465] LNetError: 411028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12351.753378] LNet: Removed LNI 192.168.202.118@tcp [12352.948517] Key type .llcrypt unregistered [12352.951539] Key type ._llcrypt unregistered [12378.231875] Key type ._llcrypt registered [12378.234949] Key type .llcrypt registered [12378.355633] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12378.703741] Key type .llcrypt unregistered [12378.705311] Key type ._llcrypt unregistered [12397.327327] Key type ._llcrypt registered [12397.329783] Key type .llcrypt registered [12397.420657] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12398.865786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12398.927820] alg: No test for adler32 (adler32-zlib) [12400.124877] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [12400.431813] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12402.143180] Key type lgssc registered [12403.275238] Lustre: Echo OBD driver; http://www.lustre.org/ [12412.187189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12412.209076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12413.872308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12419.283351] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12423.147493] LustreError: 413234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12423.167546] LustreError: 413234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12428.256871] LustreError: 413235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12429.230548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12429.800787] LustreError: 413234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12429.836764] LustreError: 413234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12429.852509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12434.928180] LustreError: 413230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12434.956195] LustreError: 413230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12437.381996] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12439.012021] LustreError: 413235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12439.033243] LustreError: 413235:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12439.035495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12445.733809] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12449.252851] LustreError: 413231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12449.287887] LustreError: 413231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12453.267195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12453.875444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12459.041232] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12474.336773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12474.357449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12474.368431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12474.377473] Lustre: Skipped 1 previous similar message [12474.392885] Lustre: Skipped 1 previous similar message [12476.474505] Lustre: server umount lustre-OST0000 complete [12481.347452] Lustre: server umount lustre-MDT0000 complete [12484.576614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12484.580316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12484.594379] LustreError: 413234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12484.621214] Lustre: Skipped 1 previous similar message [12484.645563] LustreError: 413234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12485.642855] LustreError: 413215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652918 with bad export cookie 9556674815891541284 [12485.656121] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12486.042672] Lustre: server umount lustre-MDT0001 complete [12497.882788] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12501.511516] Key type lgssc unregistered [12501.995271] LNet: 416392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12502.000453] LNetError: 416392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12502.017339] LNet: Removed LNI 192.168.202.118@tcp [12502.913726] Key type .llcrypt unregistered [12502.915536] Key type ._llcrypt unregistered [12529.460210] Key type ._llcrypt registered [12529.464637] Key type .llcrypt registered [12529.647312] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12531.399615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12531.513985] alg: No test for adler32 (adler32-zlib) [12532.718383] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [12533.121189] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12534.863233] Key type lgssc registered [12536.532524] Lustre: Echo OBD driver; http://www.lustre.org/ [12545.158977] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12545.192515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12560.737549] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12565.861592] LustreError: 417780:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.118@tcp: failed processing log, type 4: rc = -110 [12592.671603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12600.517048] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12609.599959] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12616.729500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12617.266671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12617.337328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12622.316673] LustreError: 417804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12622.346342] LustreError: 417804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12622.647644] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12627.431459] LustreError: 417805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12627.475754] LustreError: 417805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12632.491398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12632.553071] LustreError: 418816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12632.578132] LustreError: 418816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12632.950785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12638.867265] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12648.821856] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12656.562552] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12668.384910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12668.391180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12668.412477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12668.910782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12668.934796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12670.415365] Lustre: server umount lustre-OST0000 complete [12679.138486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12679.140844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12679.146290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12679.180836] Lustre: Skipped 1 previous similar message [12680.869906] Lustre: server umount lustre-MDT0000 complete [12684.262506] LustreError: 418822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12684.279852] LustreError: 418822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12685.007390] LustreError: 417788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653117 with bad export cookie 3273852809252448915 [12685.030962] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12685.034478] LustreError: 417788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12685.468824] Lustre: server umount lustre-MDT0001 complete [12696.915992] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12699.881746] Key type lgssc unregistered [12700.218501] LNet: 421351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12700.232157] LNetError: 421351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12700.251656] LNet: Removed LNI 192.168.202.118@tcp [12701.008821] Key type .llcrypt unregistered [12701.011333] Key type ._llcrypt unregistered [12725.066802] Key type ._llcrypt registered [12725.069386] Key type .llcrypt registered [12725.170791] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12726.059527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12726.099275] alg: No test for adler32 (adler32-zlib) [12727.203753] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [12727.407644] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12729.040187] Key type lgssc registered [12730.322677] Lustre: Echo OBD driver; http://www.lustre.org/ [12738.689774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12738.755576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12740.320656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12746.092238] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12750.837127] LustreError: 422766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12750.870565] LustreError: 422766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12755.605454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12756.136108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12761.474147] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12771.499578] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12779.522892] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12787.503951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12788.008049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12794.843965] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12803.009538] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12810.553438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12820.221493] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.18@tcp (stopping) [12820.963363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12820.965977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12820.985894] Lustre: Skipped 2 previous similar messages [12821.007640] Lustre: Skipped 2 previous similar messages [12825.280268] LustreError: 424017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12825.365358] Lustre: server umount lustre-MDT0000 complete [12828.307855] LustreError: 422749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.18@tcp arrived at 1788653261 with bad export cookie 6614403857256385359 [12828.332682] LustreError: 422749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12830.599903] LustreError: 422747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653263 with bad export cookie 6614403857256384694 [12830.613877] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12830.623488] LustreError: 422747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12831.205703] LustreError: 424017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12831.208875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12831.227490] LustreError: 424017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12831.274850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12831.314931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12835.232477] LustreError: 422767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12835.238465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12835.245533] LustreError: 422767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12836.990281] Lustre: server umount lustre-OST0000 complete [12840.352842] LustreError: 422766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12840.379092] LustreError: 422766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12846.560442] Lustre: server umount lustre-MDT0001 complete [12859.367489] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [12862.986265] Key type lgssc unregistered [12863.354326] LNet: 426476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12863.364213] LNetError: 426476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12863.380829] LNet: Removed LNI 192.168.202.118@tcp [12864.407667] Key type .llcrypt unregistered [12864.409749] Key type ._llcrypt unregistered [12893.662346] Key type ._llcrypt registered [12893.664856] Key type .llcrypt registered [12893.841633] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [12896.273156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12896.312265] alg: No test for adler32 (adler32-zlib) [12897.459813] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [12897.892047] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [12899.633374] Key type lgssc registered [12900.997259] Lustre: Echo OBD driver; http://www.lustre.org/ [12909.767719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12909.791745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12911.285956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12916.462854] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12921.836037] LustreError: 427890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12921.869318] LustreError: 427890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12925.444693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12925.741838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12930.690272] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12938.380644] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12944.925908] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12952.702201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12953.051530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12956.148410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12959.887823] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [12968.927404] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12975.597655] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12975.630826] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12975.640028] Lustre: Skipped 1 previous similar message [12976.382818] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [12980.707359] LustreError: 427891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12980.724963] LustreError: 427891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12980.907247] Lustre: server umount lustre-MDT0001 complete [12985.828290] LustreError: 429733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12985.862185] LustreError: 429733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12990.945681] LustreError: 427890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12990.962324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12990.986753] Lustre: Skipped 1 previous similar message [12990.994698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12995.455294] Lustre: server umount lustre-OST0000 complete [12996.073075] LustreError: 427891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13000.776851] Lustre: server umount lustre-MDT0000 complete [13015.055624] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13018.085443] Key type lgssc unregistered [13018.548728] LNet: 431577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13018.556783] LNetError: 431577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13018.575883] LNet: Removed LNI 192.168.202.118@tcp [13019.615638] Key type .llcrypt unregistered [13019.618575] Key type ._llcrypt unregistered [13046.532922] Key type ._llcrypt registered [13046.534721] Key type .llcrypt registered [13046.623173] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [13061.917115] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13062.855769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13063.056565] alg: No test for adler32 (adler32-zlib) [13064.105835] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [13064.307785] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [13065.991267] Key type lgssc registered [13067.024494] Lustre: Echo OBD driver; http://www.lustre.org/ [13109.859964] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13130.464843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13130.490509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13131.800905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13131.868885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13132.062983] Lustre: lustre-MDT0000: new disk, initializing [13132.181402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13132.230876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13137.656960] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13154.868039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13155.210421] Lustre: lustre-OST0000: new disk, initializing [13155.218198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13155.227850] Lustre: Skipped 1 previous similar message [13155.233974] Lustre: 436636:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13155.306851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13162.129594] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13165.258775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13165.271284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13165.314605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13172.193327] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13188.811120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13188.956788] Lustre: 435616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13189.020396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13189.108913] Lustre: lustre-MDT0001: new disk, initializing [13189.275592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13189.318976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13189.330930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13194.196691] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13205.050939] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13205.305399] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13209.880308] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13210.127641] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13215.973347] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [13217.257367] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13217.268558] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13217.290271] Lustre: Skipped 1 previous similar message [13217.313033] Lustre: Skipped 1 previous similar message [13221.067821] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.18@tcp (stopping) [13221.268745] Lustre: server umount lustre-MDT0001 complete [13227.490195] LustreError: 437300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13227.534337] LustreError: 437300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13232.612517] LustreError: 435624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13232.638094] LustreError: 435624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13237.730754] LustreError: 437300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13237.743267] LustreError: 437300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13242.339780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13242.352867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13242.370750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13242.861902] LustreError: 435630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13244.916313] Lustre: server umount lustre-OST0000 complete [13247.976874] LustreError: 435629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13255.910677] Lustre: server umount lustre-MDT0000 complete [13271.422224] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13274.541576] Key type lgssc unregistered [13274.845702] LNet: 440381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13274.849950] LNetError: 440381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13274.861875] LNet: Removed LNI 192.168.202.118@tcp [13275.980623] Key type .llcrypt unregistered [13275.982224] Key type ._llcrypt unregistered [13318.442680] Key type ._llcrypt registered [13318.445411] Key type .llcrypt registered [13318.538888] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [13318.792415] Key type .llcrypt unregistered [13318.795290] Key type ._llcrypt unregistered [13336.385749] Key type ._llcrypt registered [13336.387372] Key type .llcrypt registered [13336.494104] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [13338.155919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13338.231900] alg: No test for adler32 (adler32-zlib) [13339.335710] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [13339.648932] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [13341.375203] Key type lgssc registered [13342.716100] Lustre: Echo OBD driver; http://www.lustre.org/ [13370.618524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13370.661106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13371.988142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13372.020063] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13372.115312] Lustre: lustre-MDT0000: new disk, initializing [13372.237967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13372.263981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13377.932670] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13388.705561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13388.997683] Lustre: 443760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13389.048289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13389.052269] Lustre: Skipped 1 previous similar message [13389.128583] Lustre: lustre-MDT0001: new disk, initializing [13389.238981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13389.269964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13389.284610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13394.283466] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13405.600301] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13412.065450] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13420.817333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13421.095964] Lustre: lustre-OST0000: new disk, initializing [13421.100828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13421.109799] Lustre: 445731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13421.199433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13428.262860] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13430.798092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13430.804724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13430.877620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13438.138189] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13445.603654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13445.616725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13445.641528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13446.120084] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13446.142939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13446.167811] Lustre: Skipped 2 previous similar messages [13451.174543] Lustre: server umount lustre-MDT0000 complete [13455.122433] LustreError: 443754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653888 with bad export cookie 5782174645675957077 [13455.123936] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13455.132321] LustreError: 443754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13455.577613] Lustre: server umount lustre-MDT0001 complete [13469.325406] Lustre: server umount lustre-OST0000 complete [13480.383483] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:18:31 (1788653911) [13488.792933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13489.382899] LustreError: 447761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13489.399550] LustreError: 447761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13489.456233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13493.638672] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13494.754662] LustreError: 447762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13499.877055] LustreError: 447761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13502.479833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13502.926279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13507.768807] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13514.567575] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13521.174436] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13541.191480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13541.436798] Lustre: Found index 0 for lustre-OST0000, updating log [13541.615141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13548.548660] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [13558.960912] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13561.097166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13571.566556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13571.568907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13571.570138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13571.587095] Lustre: Skipped 2 previous similar messages [13574.891092] Lustre: server umount lustre-OST0000 complete [13594.603306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13594.611088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13594.629322] Lustre: Skipped 1 previous similar message [13594.650307] Lustre: Skipped 4 previous similar messages [13597.946396] Lustre: server umount lustre-MDT0000 complete [13599.713407] LustreError: 447762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13599.727951] LustreError: 447762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13601.797581] LustreError: 449602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654034 with bad export cookie 5782174645675958470 [13601.808627] LustreError: 449602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13601.811952] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13602.098288] Lustre: server umount lustre-MDT0001 complete [13603.878611] LNet: 1 local NIs in recovery (showing 1): 192.168.202.118@tcp [13611.039239] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:20:42 (1788654042) [14235.844887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14236.323325] LustreError: 454170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14236.341357] LustreError: 454170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14236.400452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14241.771785] LustreError: 454171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14242.394851] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14246.893388] LustreError: 454170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14252.095640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14252.727974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14258.407627] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14267.402846] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14274.024427] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14294.047470] LNet: 1 local NIs in recovery (showing 1): 192.168.202.118@tcp [14294.063608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14294.249666] Lustre: Found index 0 for lustre-OST0000, updating log [14294.558257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14302.044880] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14309.327518] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14311.457542] Lustre: server umount lustre-OST0000 complete [14315.188864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14315.193352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14315.208305] Lustre: Skipped 1 previous similar message [14315.224498] Lustre: Skipped 1 previous similar message [14316.256434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14316.265562] Lustre: Skipped 1 previous similar message [14317.350035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14319.400547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14319.413452] Lustre: Skipped 4 previous similar messages [14321.450491] LustreError: 454170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14321.467973] LustreError: 454170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14321.609655] Lustre: server umount lustre-MDT0000 complete [14325.541112] LustreError: 454170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14325.552508] LustreError: 454170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [14325.842680] LustreError: 456030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654758 with bad export cookie 5782174645675959555 [14325.844165] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14325.855395] LustreError: 456030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14326.242389] Lustre: server umount lustre-MDT0001 complete [14354.463229] LNet: 1 local NIs in recovery (showing 1): 192.168.202.118@tcp [14358.947621] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [14370.632932] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [14410.272796] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:34:01 (1788654841) [14432.689426] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [14436.675791] Key type lgssc unregistered [14437.109095] LNet: 460783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14437.127870] LNetError: 460783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14554.918343] LNet: Removed LNI 192.168.202.118@tcp [14556.128751] Key type .llcrypt unregistered [14556.132860] Key type ._llcrypt unregistered [14574.896524] Key type ._llcrypt registered [14574.902512] Key type .llcrypt registered [14575.065297] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [14576.429893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14576.694153] alg: No test for adler32 (adler32-zlib) [14577.887763] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [14578.311982] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [14580.056688] Key type lgssc registered [14581.365834] Lustre: Echo OBD driver; http://www.lustre.org/ [14586.210593] LNet: Added LNI 192.168.202.118@tcp100 [8/256/0/180] [14587.299972] LNet: Added LNI 192.168.202.118@tcp101 [8/256/0/180] [14588.286056] LNet: Added LNI 192.168.202.118@tcp102 [8/256/0/180] [14589.447230] LNet: Added LNI 192.168.202.118@tcp103 [8/256/0/180] [14590.396294] LNet: Added LNI 192.168.202.118@tcp104 [8/256/0/180] [14591.409790] LNet: Added LNI 192.168.202.118@tcp105 [8/256/0/180] [14592.275063] LNet: Added LNI 192.168.202.118@tcp106 [8/256/0/180] [14593.150128] LNet: Added LNI 192.168.202.118@tcp107 [8/256/0/180] [14594.064870] LNet: Added LNI 192.168.202.118@tcp108 [8/256/0/180] [14595.067555] LNet: Added LNI 192.168.202.118@tcp109 [8/256/0/180] [14596.009257] LNet: Added LNI 192.168.202.118@tcp110 [8/256/0/180] [14596.781942] LNet: Added LNI 192.168.202.118@tcp111 [8/256/0/180] [14597.757929] LNet: Added LNI 192.168.202.118@tcp112 [8/256/0/180] [14598.532150] LNet: Added LNI 192.168.202.118@tcp113 [8/256/0/180] [14599.362872] LNet: Added LNI 192.168.202.118@tcp114 [8/256/0/180] [14600.241542] LNet: Added LNI 192.168.202.118@tcp115 [8/256/0/180] [14601.109625] LNet: Added LNI 192.168.202.118@tcp116 [8/256/0/180] [14601.993932] LNet: Added LNI 192.168.202.118@tcp117 [8/256/0/180] [14602.858323] LNet: Added LNI 192.168.202.118@tcp118 [8/256/0/180] [14603.624827] LNet: Added LNI 192.168.202.118@tcp119 [8/256/0/180] [14604.483844] LNet: Added LNI 192.168.202.118@tcp120 [8/256/0/180] [14605.381855] LNet: Added LNI 192.168.202.118@tcp121 [8/256/0/180] [14606.164826] LNet: Added LNI 192.168.202.118@tcp122 [8/256/0/180] [14607.022865] LNet: Added LNI 192.168.202.118@tcp123 [8/256/0/180] [14607.879822] LNet: Added LNI 192.168.202.118@tcp124 [8/256/0/180] [14608.906918] LNet: Added LNI 192.168.202.118@tcp125 [8/256/0/180] [14609.763185] LNet: Added LNI 192.168.202.118@tcp126 [8/256/0/180] [14610.796328] LNet: Added LNI 192.168.202.118@tcp127 [8/256/0/180] [14611.729526] LNet: Added LNI 192.168.202.118@tcp128 [8/256/0/180] [14612.725488] LNet: Added LNI 192.168.202.118@tcp129 [8/256/0/180] [14613.658934] LNet: Added LNI 192.168.202.118@tcp130 [8/256/0/180] [14614.719943] LNet: Added LNI 192.168.202.118@tcp131 [8/256/0/180] [14615.775133] LNet: Added LNI 192.168.202.118@tcp132 [8/256/0/180] [14617.117652] LNet: Added LNI 192.168.202.118@tcp133 [8/256/0/180] [14618.687495] LNet: Added LNI 192.168.202.118@tcp134 [8/256/0/180] [14620.102324] LNet: Added LNI 192.168.202.118@tcp135 [8/256/0/180] [14627.149614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14627.191315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14628.391130] Lustre: 463812:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [14628.516159] LNetError: 461270:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14628.539891] LNetError: Protocol error connecting to 192.168.202.118@tcp122 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14628.562580] LNetError: Skipped 4 previous similar messages [14628.679685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14628.712585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14628.848449] Lustre: lustre-MDT0000: new disk, initializing [14628.936649] Lustre: 463867:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [14628.946710] Lustre: 463867:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14628.976316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14629.002727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14629.535459] LNet: 9 local NIs in recovery (showing 9): 192.168.202.118@tcp124, 192.168.202.118@tcp125, 192.168.202.118@tcp126, 192.168.202.118@tcp130, 192.168.202.118@tcp122, 192.168.202.118@tcp131, 192.168.202.118@tcp132, 192.168.202.118@tcp133, 192.168.202.118@tcp134 [14629.538705] LNetError: 463834:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14629.583207] LNetError: 463834:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [14629.593352] LNetError: Protocol error connecting to 192.168.202.118@tcp106 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14629.606597] LNetError: Skipped 16 previous similar messages [14631.652426] LNetError: 463837:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14631.667994] LNetError: 463837:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14631.677526] LNetError: Protocol error connecting to 192.168.202.118@tcp122 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14631.687703] LNetError: Skipped 3 previous similar messages [14633.426709] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14635.755289] LNetError: 461271:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14635.775374] LNetError: 461271:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14635.783951] LNetError: Protocol error connecting to 192.168.202.118@tcp122 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14643.941380] LNetError: 461272:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14643.953281] LNetError: Protocol error connecting to 192.168.202.118@tcp122 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14646.947247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14647.215818] Lustre: 463829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14647.386177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14647.399248] Lustre: Skipped 1 previous similar message [14647.542614] Lustre: lustre-MDT0001: new disk, initializing [14647.575610] Lustre: 464631:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [14647.586840] Lustre: 464631:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14647.686541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14647.727478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14647.745390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14653.008846] Lustre: 464833:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [14653.029145] Lustre: 464833:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14654.184571] LNetError: 463838:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14654.234392] LNetError: 463838:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14654.244655] LNetError: Protocol error connecting to 192.168.202.118@tcp135 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14654.258878] LNetError: Skipped 3 previous similar messages [14654.293687] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14665.746486] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14671.523393] LNetError: 463837:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14671.532516] LNetError: 463837:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [14671.536646] LNetError: Protocol error connecting to 192.168.202.118@tcp120 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14671.541706] LNetError: Skipped 13 previous similar messages [14672.427271] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14680.131641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14680.356421] Lustre: lustre-OST0000: new disk, initializing [14680.359261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14680.364890] Lustre: 465805:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14680.436544] Lustre: 465810:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.118@tcp: can't add NID 192.168.202.118@tcp130: rc = -75 [14680.444198] Lustre: 465810:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14680.483790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14685.861453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14685.894825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14686.229930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14689.036794] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14690.022545] LNet: 2 local NIs in recovery (showing 2): 192.168.202.118@tcp135, 192.168.202.118@tcp122 [14701.111186] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14705.376547] LNetError: 461272:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.118 [14705.392316] LNetError: 461272:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 30 previous similar messages [14705.407377] LNetError: Protocol error connecting to 192.168.202.118@tcp127 at host 192.168.202.118:988: is it running a compatible version of Lustre? [14705.419071] LNetError: Skipped 30 previous similar messages [14711.776988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14711.779856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14711.786858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14711.798795] LustreError: Skipped 1 previous similar message [14711.838206] Lustre: Skipped 1 previous similar message [14716.247731] Lustre: server umount lustre-OST0000 complete [14722.023553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14722.031034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14722.044035] Lustre: Skipped 1 previous similar message [14722.048941] Lustre: Skipped 1 previous similar message [14726.444610] Lustre: server umount lustre-MDT0000 complete [14730.879524] LustreError: 464630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655163 with bad export cookie 3229799924390712576 [14730.892717] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14730.908672] LustreError: 464630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14731.616766] Lustre: server umount lustre-MDT0001 complete [14745.911353] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [14750.428541] Key type lgssc unregistered [14750.953278] LNet: 467718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14750.975791] LNetError: 467718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14750.991736] LNet: Removed LNI 192.168.202.118@tcp [14752.031239] LNet: Removed LNI 192.168.202.118@tcp100 [14752.033543] LNetError: 467718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14752.051079] LNetError: 467718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14752.072397] LNet: Removed LNI 192.168.202.118@tcp101 [14753.120162] LNet: Removed LNI 192.168.202.118@tcp102 [14753.124226] LNetError: 467718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14753.131466] LNetError: 467718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14753.136632] LNet: Removed LNI 192.168.202.118@tcp103 [14754.143364] LNet: Removed LNI 192.168.202.118@tcp104 [14755.167169] LNet: Removed LNI 192.168.202.118@tcp105 [14755.176871] LNetError: 467718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14755.196058] LNetError: 467718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14755.213126] LNet: Removed LNI 192.168.202.118@tcp106 [14756.257714] LNet: Removed LNI 192.168.202.118@tcp107 [14757.279913] LNet: Removed LNI 192.168.202.118@tcp108 [14758.304601] LNet: Removed LNI 192.168.202.118@tcp109 [14759.327637] LNet: Removed LNI 192.168.202.118@tcp110 [14759.329818] LNetError: 467718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14759.340954] LNetError: 467718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14759.351235] LNet: Removed LNI 192.168.202.118@tcp111 [14760.416725] LNet: Removed LNI 192.168.202.118@tcp112 [14761.439286] LNet: Removed LNI 192.168.202.118@tcp113 [14762.463193] LNet: Removed LNI 192.168.202.118@tcp114 [14763.488174] LNet: Removed LNI 192.168.202.118@tcp115 [14764.511440] LNet: Removed LNI 192.168.202.118@tcp116 [14765.535940] LNet: Removed LNI 192.168.202.118@tcp117 [14766.559914] LNet: Removed LNI 192.168.202.118@tcp118 [14767.583718] LNet: Removed LNI 192.168.202.118@tcp119 [14767.589262] LNetError: 467718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14767.601417] LNetError: 467718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14767.613991] LNet: Removed LNI 192.168.202.118@tcp120 [14768.672536] LNet: Removed LNI 192.168.202.118@tcp121 [14768.684335] LNet: Removed LNI 192.168.202.118@tcp122 [14769.699652] LNet: Removed LNI 192.168.202.118@tcp123 [14770.720259] LNet: Removed LNI 192.168.202.118@tcp124 [14771.743274] LNet: Removed LNI 192.168.202.118@tcp125 [14772.767178] LNet: Removed LNI 192.168.202.118@tcp126 [14773.791549] LNet: Removed LNI 192.168.202.118@tcp127 [14774.817166] LNet: Removed LNI 192.168.202.118@tcp128 [14775.839509] LNet: Removed LNI 192.168.202.118@tcp129 [14776.863507] LNet: Removed LNI 192.168.202.118@tcp130 [14777.887669] LNet: Removed LNI 192.168.202.118@tcp131 [14778.911184] LNet: Removed LNI 192.168.202.118@tcp132 [14779.935449] LNet: Removed LNI 192.168.202.118@tcp133 [14780.959329] LNet: Removed LNI 192.168.202.118@tcp134 [14780.972872] LNet: Removed LNI 192.168.202.118@tcp135 [14781.982630] Key type .llcrypt unregistered [14781.984410] Key type ._llcrypt unregistered [14808.537513] Key type ._llcrypt registered [14808.541832] Key type .llcrypt registered [14808.665302] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [14825.491515] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [14826.751434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14826.951171] alg: No test for adler32 (adler32-zlib) [14828.005976] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [14828.224367] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [14829.879225] Key type lgssc registered [14831.457602] Lustre: Echo OBD driver; http://www.lustre.org/ [14869.735478] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:41:40 (1788655300) [14888.965862] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [14892.445340] Key type lgssc unregistered [14892.728583] LNet: 471822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14892.734766] LNetError: 471822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14892.750298] LNet: Removed LNI 192.168.202.118@tcp [14893.500805] Key type .llcrypt unregistered [14893.503688] Key type ._llcrypt unregistered [14915.161779] Key type ._llcrypt registered [14915.164467] Key type .llcrypt registered [14915.436895] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [14917.108646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14917.163628] alg: No test for adler32 (adler32-zlib) [14918.829495] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [14919.204816] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [14921.047229] Key type lgssc registered [14923.001044] Lustre: Echo OBD driver; http://www.lustre.org/ [14936.225465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14936.273147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14937.677045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14937.724879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14937.841590] Lustre: lustre-MDT0000: new disk, initializing [14937.941623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14937.995077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14943.513991] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14957.809484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14958.048709] Lustre: 473031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14958.098705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14958.104518] Lustre: Skipped 1 previous similar message [14958.241320] Lustre: lustre-MDT0001: new disk, initializing [14958.390102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14958.479847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14958.501237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14963.686274] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14974.456412] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14980.542636] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14987.854182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14988.128695] Lustre: lustre-OST0000: new disk, initializing [14988.134663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14988.148549] Lustre: 475002:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14988.310155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14995.878776] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [14997.083251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14997.127937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14997.377138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15007.529288] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15013.370781] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing lnet_if_list [15019.452497] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15019.465640] LNet: Added LNI 192.168.202.118@tcp10 [8/256/0/180] [15026.199662] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15026.238844] LNet: Added LNI 192.168.202.118@tcp11 [8/256/0/180] [15031.264915] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15031.295793] LNet: Added LNI 192.168.202.118@tcp12 [8/256/0/180] [15036.398380] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15036.461306] LNet: Added LNI 192.168.202.118@tcp13 [8/256/0/180] [15042.064828] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15042.103376] LNet: Added LNI 192.168.202.118@tcp14 [8/256/0/180] [15047.106253] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15047.148570] LNet: Added LNI 192.168.202.118@tcp15 [8/256/0/180] [15052.787848] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15052.837742] LNet: Added LNI 192.168.202.118@tcp16 [8/256/0/180] [15058.706761] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15058.729117] LNet: Added LNI 192.168.202.118@tcp17 [8/256/0/180] [15066.181390] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15066.220802] LNet: Added LNI 192.168.202.118@tcp18 [8/256/0/180] [15073.519258] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15073.553829] LNet: Added LNI 192.168.202.118@tcp19 [8/256/0/180] [15079.201830] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15079.260238] LNet: Added LNI 192.168.202.118@tcp20 [8/256/0/180] [15085.079329] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15085.104614] LNet: Added LNI 192.168.202.118@tcp21 [8/256/0/180] [15090.949958] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15090.989397] LNet: Added LNI 192.168.202.118@tcp22 [8/256/0/180] [15096.514440] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15096.566942] LNet: Added LNI 192.168.202.118@tcp23 [8/256/0/180] [15102.625892] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15102.645862] LNet: Added LNI 192.168.202.118@tcp24 [8/256/0/180] [15108.179633] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15108.198619] LNet: Added LNI 192.168.202.118@tcp25 [8/256/0/180] [15114.138647] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15114.158772] LNet: Added LNI 192.168.202.118@tcp26 [8/256/0/180] [15120.095830] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15120.136812] LNet: Added LNI 192.168.202.118@tcp27 [8/256/0/180] [15125.589392] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15125.629710] LNet: Added LNI 192.168.202.118@tcp28 [8/256/0/180] [15131.605969] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15131.632887] LNet: Added LNI 192.168.202.118@tcp29 [8/256/0/180] [15138.776580] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15138.818956] LNet: Added LNI 192.168.202.118@tcp30 [8/256/0/180] [15145.750660] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15145.781175] LNet: Added LNI 192.168.202.118@tcp31 [8/256/0/180] [15152.712835] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15152.750072] LNet: Added LNI 192.168.202.118@tcp32 [8/256/0/180] [15160.053629] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15160.074132] LNet: Added LNI 192.168.202.118@tcp33 [8/256/0/180] [15167.970221] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15168.002780] LNet: Added LNI 192.168.202.118@tcp34 [8/256/0/180] [15174.554887] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15174.589352] LNet: Added LNI 192.168.202.118@tcp35 [8/256/0/180] [15182.021360] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15182.066163] LNet: Added LNI 192.168.202.118@tcp36 [8/256/0/180] [15189.068505] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15189.118368] LNet: Added LNI 192.168.202.118@tcp37 [8/256/0/180] [15197.723648] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15197.769200] LNet: Added LNI 192.168.202.118@tcp38 [8/256/0/180] [15205.043879] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15205.084497] LNet: Added LNI 192.168.202.118@tcp39 [8/256/0/180] [15213.836215] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15213.868683] LNet: Added LNI 192.168.202.118@tcp40 [8/256/0/180] [15221.937296] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15221.959592] LNet: Added LNI 192.168.202.118@tcp41 [8/256/0/180] [15229.987277] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15230.014944] LNet: Added LNI 192.168.202.118@tcp42 [8/256/0/180] [15237.188990] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15237.235080] LNet: Added LNI 192.168.202.118@tcp43 [8/256/0/180] [15244.451485] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15244.477282] LNet: Added LNI 192.168.202.118@tcp44 [8/256/0/180] [15251.681268] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15251.713674] LNet: Added LNI 192.168.202.118@tcp45 [8/256/0/180] [15260.089708] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15260.125242] LNet: Added LNI 192.168.202.118@tcp46 [8/256/0/180] [15268.062243] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15268.094419] LNet: Added LNI 192.168.202.118@tcp47 [8/256/0/180] [15277.280985] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15277.323490] LNet: Added LNI 192.168.202.118@tcp48 [8/256/0/180] [15287.517576] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15287.569400] LNet: Added LNI 192.168.202.118@tcp49 [8/256/0/180] [15299.560644] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15299.583531] LNet: Added LNI 192.168.202.118@tcp50 [8/256/0/180] [15320.039672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15320.055887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15320.067940] Lustre: Skipped 1 previous similar message [15320.070406] Lustre: Skipped 2 previous similar messages [15325.166513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15325.176150] Lustre: Skipped 3 previous similar messages [15325.528940] Lustre: server umount lustre-MDT0000 complete [15328.926315] LustreError: 473836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655761 with bad export cookie 8109702545930112729 [15328.927152] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15328.938290] LustreError: 473836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15329.342963] Lustre: server umount lustre-MDT0001 complete [15346.656986] Lustre: 472304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788655763/real 1788655763] req@000000008640db35 x1875541074548352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788655779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15346.691300] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15348.724328] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [15349.473802] Lustre: 472303:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788655766/real 1788655766] req@00000000c0d75914 x1875541074548480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788655782 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15351.839108] Lustre: 472301:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788655768/real 1788655768] req@000000007602739f x1875541074548608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788655784 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15354.925202] Lustre: 472301:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788655771/real 1788655771] req@0000000004faee60 x1875541074548864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788655787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15358.811818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15359.971183] Lustre: 472304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788655776/real 1788655776] req@00000000aa4ca91b x1875541074548992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788655792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15373.343700] LustreError: 472300:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000755dbe45 x1875541074549760/t0(0) o250->MGC192.168.202.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15373.673829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15375.713762] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15378.304537] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15381.985773] LustreError: 474995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15382.000637] LustreError: 474995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15387.107019] LustreError: 474996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15387.462255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15388.094445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15391.159456] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15393.596579] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15403.552665] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15410.068022] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15429.088322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15429.103323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15429.125704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15430.114258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15432.164111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15432.167151] Lustre: Skipped 1 previous similar message [15437.284309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15437.296744] Lustre: Skipped 1 previous similar message [15442.406338] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15442.808800] Lustre: server umount lustre-OST0000 complete [15449.569098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15449.581060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15449.590568] Lustre: Skipped 1 previous similar message [15449.605639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15449.614506] Lustre: Skipped 2 previous similar messages [15462.368940] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15462.532911] Lustre: server umount lustre-MDT0000 complete [15462.893390] LustreError: 499249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15462.904984] LustreError: 499249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15467.183870] LustreError: 473836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655900 with bad export cookie 8109702545930115270 [15467.195673] LustreError: 473836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15467.199537] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15467.740930] Lustre: server umount lustre-MDT0001 complete [15483.733513] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [15488.617651] Key type lgssc unregistered [15489.248558] LNet: 502028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15489.263842] LNetError: 502028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15489.275860] LNet: Removed LNI 192.168.202.118@tcp [15489.284913] LNet: Removed LNI 192.168.202.118@tcp10 [15489.291832] LNet: Removed LNI 192.168.202.118@tcp11 [15489.298619] LNet: Removed LNI 192.168.202.118@tcp12 [15489.303738] LNet: Removed LNI 192.168.202.118@tcp13 [15489.310087] LNet: Removed LNI 192.168.202.118@tcp14 [15489.312427] LNet: Removed LNI 192.168.202.118@tcp15 [15489.324025] LNet: Removed LNI 192.168.202.118@tcp16 [15489.332242] LNet: Removed LNI 192.168.202.118@tcp17 [15489.334061] LNet: Removed LNI 192.168.202.118@tcp18 [15489.339474] LNet: Removed LNI 192.168.202.118@tcp19 [15489.346980] LNet: Removed LNI 192.168.202.118@tcp20 [15489.354957] LNet: Removed LNI 192.168.202.118@tcp21 [15489.362290] LNet: Removed LNI 192.168.202.118@tcp22 [15489.364320] LNet: Removed LNI 192.168.202.118@tcp23 [15489.366309] LNet: Removed LNI 192.168.202.118@tcp24 [15489.375083] LNet: Removed LNI 192.168.202.118@tcp25 [15489.381408] LNet: Removed LNI 192.168.202.118@tcp26 [15489.396686] LNet: Removed LNI 192.168.202.118@tcp27 [15489.398797] LNet: Removed LNI 192.168.202.118@tcp28 [15489.400430] LNet: Removed LNI 192.168.202.118@tcp29 [15489.408560] LNet: Removed LNI 192.168.202.118@tcp30 [15489.422712] LNet: Removed LNI 192.168.202.118@tcp31 [15489.431808] LNet: Removed LNI 192.168.202.118@tcp32 [15489.466780] LNet: Removed LNI 192.168.202.118@tcp33 [15489.468656] LNet: Removed LNI 192.168.202.118@tcp34 [15489.473611] LNet: Removed LNI 192.168.202.118@tcp35 [15489.483755] LNet: Removed LNI 192.168.202.118@tcp36 [15489.488806] LNet: Removed LNI 192.168.202.118@tcp37 [15489.491148] LNet: Removed LNI 192.168.202.118@tcp38 [15489.499374] LNet: Removed LNI 192.168.202.118@tcp39 [15489.504988] LNet: Removed LNI 192.168.202.118@tcp40 [15489.507892] LNet: Removed LNI 192.168.202.118@tcp41 [15489.512409] LNet: Removed LNI 192.168.202.118@tcp42 [15489.521317] LNet: Removed LNI 192.168.202.118@tcp43 [15489.524456] LNet: Removed LNI 192.168.202.118@tcp44 [15489.526867] LNet: Removed LNI 192.168.202.118@tcp45 [15489.529676] LNet: Removed LNI 192.168.202.118@tcp46 [15489.532515] LNet: Removed LNI 192.168.202.118@tcp47 [15489.535362] LNet: Removed LNI 192.168.202.118@tcp48 [15489.537462] LNet: Removed LNI 192.168.202.118@tcp49 [15489.545980] LNet: Removed LNI 192.168.202.118@tcp50 [15491.033860] Key type .llcrypt unregistered [15491.041909] Key type ._llcrypt unregistered [15513.931535] Key type ._llcrypt registered [15513.936694] Key type .llcrypt registered [15514.047802] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [15528.686324] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [15530.469575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15530.519341] alg: No test for adler32 (adler32-zlib) [15531.608537] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [15531.828782] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [15533.495420] Key type lgssc registered [15534.576353] Lustre: Echo OBD driver; http://www.lustre.org/ [15567.833741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15567.889303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15569.146664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15569.175294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15569.329298] Lustre: lustre-MDT0000: new disk, initializing [15569.446401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15569.463379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15574.369159] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15584.547024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15584.649662] Lustre: 505220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15584.699707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15584.701779] Lustre: Skipped 1 previous similar message [15584.788753] Lustre: lustre-MDT0001: new disk, initializing [15584.898691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15584.936540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15584.957476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15589.634852] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15600.183298] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15605.960404] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15613.493326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15613.756056] Lustre: *** cfs_fail_loc=90f, val=0*** [15613.867910] Lustre: lustre-OST0000: new disk, initializing [15613.874915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15613.888743] Lustre: 507240:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15613.994163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15620.670259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15620.690526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15620.762463] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15620.844046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15630.481647] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15641.569817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15641.573229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15641.592726] Lustre: Skipped 1 previous similar message [15641.593454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15641.628937] Lustre: Skipped 1 previous similar message [15646.697529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15646.712170] Lustre: Skipped 1 previous similar message [15651.297965] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15651.444971] Lustre: server umount lustre-OST0000 complete [15656.427322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15656.435411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15656.451052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15660.994708] Lustre: server umount lustre-MDT0000 complete [15664.237038] LustreError: 505215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656097 with bad export cookie 13650882635054978791 [15664.245182] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15664.248806] LustreError: 505215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15664.581942] Lustre: server umount lustre-MDT0001 complete [15675.579601] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [15678.237753] Key type lgssc unregistered [15678.556091] LNet: 509146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15678.564382] LNetError: 509146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15678.580990] LNet: Removed LNI 192.168.202.118@tcp [15679.637648] Key type .llcrypt unregistered [15679.647513] Key type ._llcrypt unregistered [15698.925834] Key type ._llcrypt registered [15698.927866] Key type .llcrypt registered [15699.029321] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [15700.002095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15700.134205] alg: No test for adler32 (adler32-zlib) [15701.269612] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [15701.566370] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [15703.247469] Key type lgssc registered [15704.649305] Lustre: Echo OBD driver; http://www.lustre.org/ [15711.116463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15711.133740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15712.597906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15717.388107] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15723.006520] LustreError: 510417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15723.033658] LustreError: 510417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15725.885756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15726.337964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15730.440328] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15737.745935] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15743.639954] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15751.340852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15752.110164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15759.406392] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15767.036547] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15781.195760] loop0: detected capacity change from 0 to 204800000 [15781.367508] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15781.374136] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15781.398103] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15787.207356] loop0: detected capacity change from 0 to 204800000 [15787.474680] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15787.486581] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15787.501834] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15792.819356] loop0: detected capacity change from 0 to 204800000 [15799.300587] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15799.491352] Lustre: fs15246-OST0000: new disk, initializing [15799.496050] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15799.509262] Lustre: 514008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15799.567256] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15806.406661] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15817.755868] loop1: detected capacity change from 0 to 204800000 [15823.431705] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15823.523022] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15823.568498] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15823.624526] Lustre: fs15246-MDT0000: new disk, initializing [15823.758421] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15823.780721] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15823.845629] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15824.933517] Lustre: 515131:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000007cede805 x1875541894414592/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788656273 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15827.545330] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [15834.018309] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15834.032107] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15834.093441] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15852.534511] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15852.564106] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15857.656179] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15858.021371] Lustre: server umount fs15246-MDT0000 complete [15862.758224] LustreError: 510412:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15867.876621] LustreError: 510412:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15868.662312] Lustre: server umount fs15246-OST0000 complete [15880.163502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15880.175739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15880.201842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15881.190715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15885.823921] Lustre: server umount lustre-OST0000 complete [15890.400658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15890.417412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15890.442381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15890.456749] Lustre: Skipped 2 previous similar messages [15896.544853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15896.556191] Lustre: Skipped 2 previous similar messages [15904.223153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15904.501227] Lustre: server umount lustre-MDT0000 complete [15906.796947] LustreError: 512905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15906.834939] LustreError: 512905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15909.330174] LustreError: 510399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656342 with bad export cookie 6022499097542081053 [15909.335051] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15909.343692] LustreError: 510399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15909.867384] Lustre: server umount lustre-MDT0001 complete [15923.656989] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [15927.326604] Key type lgssc unregistered [15927.655592] LNet: 518230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15927.663270] LNetError: 518230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15927.680774] LNet: Removed LNI 192.168.202.118@tcp [15928.708461] Key type .llcrypt unregistered [15928.712571] Key type ._llcrypt unregistered [15946.977493] Key type ._llcrypt registered [15946.979993] Key type .llcrypt registered [15947.076302] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [15948.042388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15948.142922] alg: No test for adler32 (adler32-zlib) [15949.144890] Lustre: Lustre: Build Version: 2.17.58_3_g980e1e2 [15949.365777] LNet: Added LNI 192.168.202.118@tcp [8/256/0/180] [15951.024280] Key type lgssc registered [15952.069694] Lustre: Echo OBD driver; http://www.lustre.org/ [15986.888774] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [15996.062982] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [16036.229453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16036.249377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16037.553847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16037.630336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16037.778260] Lustre: lustre-MDT0000: new disk, initializing [16037.979522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16038.008748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16042.490900] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16053.123329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16053.315923] Lustre: 522708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16053.378491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16053.383857] Lustre: Skipped 1 previous similar message [16053.471368] Lustre: lustre-MDT0001: new disk, initializing [16053.610315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16053.679267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16053.695740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16058.875894] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16067.683919] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16074.191507] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16081.274468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16081.603176] Lustre: lustre-OST0000: new disk, initializing [16081.608863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16081.616821] Lustre: 524676:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16081.699900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16087.605594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16087.621702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16087.753507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16088.412484] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_default_debug -1 all [16097.090493] Lustre: DEBUG MARKER: oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16101.356960] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16101.546665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16105.716212] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16106.037834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16108.515963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16108.518658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16108.521472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16108.537559] Lustre: Skipped 1 previous similar message [16108.548550] Lustre: Skipped 1 previous similar message [16113.636600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16113.646646] Lustre: Skipped 1 previous similar message [16118.776849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16118.786336] Lustre: Skipped 1 previous similar message [16122.335211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16122.671175] Lustre: server umount lustre-OST0000 complete [16128.996688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16129.009107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16134.112977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16134.120000] Lustre: Skipped 2 previous similar messages [16141.280681] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16141.508811] Lustre: server umount lustre-MDT0000 complete [16146.119473] LustreError: 524687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656579 with bad export cookie 6048151720696839948 [16146.124334] LustreError: MGC192.168.202.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16146.130792] LustreError: 524687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16146.334961] Lustre: server umount lustre-MDT0001 complete [16174.469379] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing set_hostid [16184.396363] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing load_modules_local [16217.700660] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15915 sec ========= 21:04:09 (1788656649) [16219.364533] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:04:10 (1788656650) === [16222.548348] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:04:13 (1788656653) === [16261.706764] Lustre: DEBUG MARKER: oleg218-server.virtnet: executing unload_modules_local [16264.460931] Key type lgssc unregistered [16264.756674] LNet: 531322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16264.766463] LNetError: 531322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16264.789794] LNet: Removed LNI 192.168.202.118@tcp [16265.893947] Key type .llcrypt unregistered [16265.898855] Key type ._llcrypt unregistered