[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480216136 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002331] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006861] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009135] LSM: Security Framework initializing [ 0.010055] Yama: becoming mindful. [ 0.012059] SELinux: Initializing. [ 0.013060] *** VALIDATE selinux *** [ 0.022134] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027412] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032022] *** VALIDATE tmpfs *** [ 0.034039] *** VALIDATE proc *** [ 0.035262] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044463] debug: unmapping init [mem 0xffffffff88a59000-0xffffffff88a60fff] [ 0.046176] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047721] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048022] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050009] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052013] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055305] rcu: Hierarchical SRCU implementation. [ 0.057410] smp: Bringing up secondary CPUs ... [ 0.058571] x86: Booting SMP configuration: [ 0.059027] .... node #0, CPUs: #1 #2 #3 [ 0.062634] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.213124] node 0 deferred pages initialised in 146ms [ 0.217140] devtmpfs: initialized [ 0.218359] x86/mm: Memory block size: 128MB [ 0.222000] gcov: version magic: 0x41383552 [ 0.226346] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.230115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.232417] pinctrl core: initialized pinctrl subsystem [ 0.234231] [ 0.234786] ************************************************************* [ 0.238023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.240017] ** ** [ 0.242017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.244024] ** ** [ 0.247020] ** This means that this kernel is built to expose internal ** [ 0.250023] ** IOMMU data structures, which may compromise security on ** [ 0.252020] ** your system. ** [ 0.254020] ** ** [ 0.257030] ** If you see this message and you are not debugging the ** [ 0.259048] ** kernel, report this immediately to your vendor! ** [ 0.262031] ** ** [ 0.265019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267046] ************************************************************* [ 0.270873] NET: Registered protocol family 16 [ 0.273514] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.276064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.279061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.282158] cpuidle: using governor menu [ 0.283911] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.284413] PCI: Using configuration type 1 for base access [ 0.285122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.292119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.295043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.298076] cryptd: max_cpu_qlen set to 1000 [ 0.301244] ACPI: Added _OSI(Module Device) [ 0.302012] ACPI: Added _OSI(Processor Device) [ 0.303014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.305011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.309299] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.315411] ACPI: Interpreter enabled [ 0.316080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.318036] ACPI: Using IOAPIC for interrupt routing [ 0.320177] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.324486] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.336342] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.339068] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.342035] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.346121] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.352449] acpiphp: Slot [2] registered [ 0.354229] acpiphp: Slot [5] registered [ 0.356208] acpiphp: Slot [6] registered [ 0.357217] acpiphp: Slot [7] registered [ 0.359183] acpiphp: Slot [8] registered [ 0.361145] acpiphp: Slot [9] registered [ 0.362189] acpiphp: Slot [10] registered [ 0.364171] acpiphp: Slot [3] registered [ 0.365145] acpiphp: Slot [4] registered [ 0.367159] acpiphp: Slot [11] registered [ 0.369206] acpiphp: Slot [12] registered [ 0.371156] acpiphp: Slot [13] registered [ 0.372156] acpiphp: Slot [14] registered [ 0.374330] acpiphp: Slot [15] registered [ 0.376143] acpiphp: Slot [16] registered [ 0.378155] acpiphp: Slot [17] registered [ 0.380203] acpiphp: Slot [18] registered [ 0.381173] acpiphp: Slot [19] registered [ 0.383212] acpiphp: Slot [20] registered [ 0.385147] acpiphp: Slot [21] registered [ 0.387139] acpiphp: Slot [22] registered [ 0.388146] acpiphp: Slot [23] registered [ 0.390153] acpiphp: Slot [24] registered [ 0.392125] acpiphp: Slot [25] registered [ 0.393127] acpiphp: Slot [26] registered [ 0.395122] acpiphp: Slot [27] registered [ 0.397117] acpiphp: Slot [28] registered [ 0.399116] acpiphp: Slot [29] registered [ 0.400103] acpiphp: Slot [30] registered [ 0.402128] acpiphp: Slot [31] registered [ 0.403058] PCI host bridge to bus 0000:00 [ 0.404020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.406024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.409022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.411024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.414025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.417037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.419254] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.423304] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.427493] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.437923] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.442919] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.446032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.448050] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.451038] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.454619] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.457826] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.461047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.465027] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.470014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.484900] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.491014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.496820] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.504014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.512013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.536026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.548227] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.556016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.563013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.577016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.590841] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.598068] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.607022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.628029] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.641312] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.648016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.656024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.674025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.690412] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.697018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.703020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.718023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.727800] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.735022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.741028] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.760021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.769850] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.772381] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.775446] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.778401] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.781259] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.786154] iommu: Default domain type: Passthrough [ 0.789512] SCSI subsystem initialized [ 0.791169] ACPI: bus type USB registered [ 0.792115] usbcore: registered new interface driver usbfs [ 0.795098] usbcore: registered new interface driver hub [ 0.797083] usbcore: registered new device driver usb [ 0.799156] pps_core: LinuxPPS API ver. 1 registered [ 0.801014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.805061] PTP clock support registered [ 0.807139] EDAC MC: Ver: 3.0.0 [ 0.809124] PCI: Using ACPI for IRQ routing [ 0.811831] NetLabel: Initializing [ 0.813013] NetLabel: domain hash size = 128 [ 0.815011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.817079] NetLabel: unlabeled traffic allowed by default [ 0.819135] vgaarb: loaded [ 0.821249] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.823016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.829014] clocksource: Switched to clocksource kvm-clock [ 0.937450] VFS: Disk quotas dquot_6.6.0 [ 0.939722] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.943734] *** VALIDATE ramfs *** [ 0.945611] *** VALIDATE hugetlbfs *** [ 0.947823] pnp: PnP ACPI init [ 0.950454] pnp: PnP ACPI: found 6 devices [ 0.977808] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.981471] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.983724] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.985988] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.988461] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.991116] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.995393] NET: Registered protocol family 2 [ 0.997716] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.002921] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.006752] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.011101] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.015217] TCP: Hash tables configured (established 65536 bind 65536) [ 1.018302] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.021636] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.024215] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.026643] NET: Registered protocol family 1 [ 1.030071] RPC: Registered named UNIX socket transport module. [ 1.032774] RPC: Registered udp transport module. [ 1.034695] RPC: Registered tcp transport module. [ 1.037639] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.041519] NET: Registered protocol family 44 [ 1.043410] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.045870] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.048282] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.050065] PCI: CLS 0 bytes, default 64 [ 1.051352] Unpacking initramfs... [ 2.455337] debug: unmapping init [mem 0xffff8d483cc54000-0xffff8d483ffbffff] [ 2.459648] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.462187] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.465204] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.987212] Initialise system trusted keyrings [ 2.989024] Key type blacklist registered [ 2.995513] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.006046] zbud: loaded [ 3.012124] *** VALIDATE nfs *** [ 3.013631] *** VALIDATE nfs4 *** [ 3.016735] pstore: using deflate compression [ 3.019470] Platform Keyring initialized [ 3.137843] NET: Registered protocol family 38 [ 3.139916] Key type asymmetric registered [ 3.141838] Asymmetric key parser 'x509' registered [ 3.144094] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.148151] io scheduler mq-deadline registered [ 3.150027] io scheduler kyber registered [ 3.151925] io scheduler bfq registered [ 3.154141] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.157679] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.161421] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.164682] ACPI: Power Button [PWRF] [ 3.173337] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.182618] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.197135] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.205685] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.225212] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.254995] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.285626] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.291922] Non-volatile memory driver v1.3 [ 3.293924] Linux agpgart interface v0.103 [ 3.329482] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.333646] vda: detected capacity change from 0 to 76677120 [ 3.348829] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.351649] vdb: detected capacity change from 0 to 1073741824 [ 3.365609] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.368302] vdc: detected capacity change from 0 to 2621440000 [ 3.384077] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.387704] vdd: detected capacity change from 0 to 2621440000 [ 3.404784] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.407584] vde: detected capacity change from 0 to 4294967296 [ 3.433693] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.437212] vdf: detected capacity change from 0 to 4294967296 [ 3.448965] libphy: Fixed MDIO Bus: probed [ 3.455535] usbcore: registered new interface driver usbserial_generic [ 3.457909] usbserial: USB Serial support registered for generic [ 3.460503] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.466483] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.468923] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.472164] mousedev: PS/2 mouse device common for all mice [ 3.475666] rtc_cmos 00:05: RTC can wake from S4 [ 3.478873] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.484208] rtc_cmos 00:05: registered as rtc0 [ 3.486774] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.487510] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.489673] intel_pstate: CPU model not supported [ 3.497961] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.500535] hid: raw HID events driver (C) Jiri Kosina [ 3.504545] usbcore: registered new interface driver usbhid [ 3.506410] usbhid: USB HID core driver [ 3.507760] drop_monitor: Initializing network drop monitor service [ 3.511365] Initializing XFRM netlink socket [ 3.513260] NET: Registered protocol family 10 [ 3.516401] Segment Routing with IPv6 [ 3.518049] NET: Registered protocol family 17 [ 3.520168] mpls_gso: MPLS GSO support [ 3.526329] RAS: Correctable Errors collector initialized. [ 3.528591] AVX version of gcm_enc/dec engaged. [ 3.530379] AES CTR mode by8 optimization enabled [ 3.625950] sched_clock: Marking stable (3625903988, 0)->(4532264544, -906360556) [ 3.628622] registered taskstats version 1 [ 3.631341] Loading compiled-in X.509 certificates [ 3.633439] zswap: loaded using pool lzo/zbud [ 3.660258] Key type big_key registered [ 3.673133] Key type encrypted registered [ 3.674551] ima: No TPM chip found, activating TPM-bypass! [ 3.676902] ima: Allocated hash algorithm: sha1 [ 3.678453] ima: No architecture policies found [ 3.680075] evm: Initialising EVM extended attributes: [ 3.682091] evm: security.selinux [ 3.683354] evm: security.ima [ 3.684185] evm: security.capability [ 3.685219] evm: HMAC attrs: 0x1 [ 3.687275] rtc_cmos 00:05: setting system clock to 2026-09-03 12:30:36 UTC (1788438636) [ 3.694076] debug: unmapping init [mem 0xffffffff89a03000-0xffffffff89bfffff] [ 3.697266] debug: unmapping init [mem 0xffffffff88782000-0xffffffff88a58fff] [ 3.710075] Write protecting the kernel read-only data: 28672k [ 3.713573] debug: unmapping init [mem 0xffffffff86e03000-0xffffffff86ffffff] [ 3.716697] debug: unmapping init [mem 0xffffffff87714000-0xffffffff877fffff] [ 3.755604] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.765251] systemd[1]: Detected virtualization kvm. [ 3.767388] systemd[1]: Detected architecture x86-64. [ 3.769386] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.795381] systemd[1]: No hostname configured. [ 3.797199] systemd[1]: Set hostname to . [ 3.799479] random: systemd: uninitialized urandom read (16 bytes read) [ 3.802198] systemd[1]: Initializing machine ID from random generator. [ 3.854835] random: ln: uninitialized urandom read (6 bytes read) [ 3.953901] random: systemd: uninitialized urandom read (16 bytes read) [ 3.957231] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.963550] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.972943] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.647474] device-mapper: uevent: version 1.0.3 [ 4.649638] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ 4.976242] random: fast init done Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.461269] virtio_net virtio0 ens2: renamed from eth0 [ 5.519751] scsi host0: ata_piix [ 5.567330] scsi host1: ata_piix [ 5.569284] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.572250] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.291762] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.189405] random: crng init done [ 10.190984] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.696687] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.887221] printk: systemd: 23 output lines suppressed due to ratelimiting [ 12.154815] SELinux: Disabled at runtime. [ 12.224194] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.232151] systemd[1]: Detected virtualization kvm. [ 12.233939] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.749041] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.752512] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.761591] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.766143] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.770282] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.779436] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.799515] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.853280] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.283509] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.551816] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.654158] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.729334] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.745917] EDAC sbridge: Ver: 1.1.2 [ 15.445618] Key type dns_resolver registered [ 15.760236] NFS: Registering the id_resolver key type [ 15.762132] Key type id_resolver registered [ 15.763955] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg331-server login: [ 52.844478] libcfs: loading out-of-tree module taints kernel. [ 52.920399] Key type ._llcrypt registered [ 52.924314] Key type .llcrypt registered [ 53.064844] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 72.636913] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 74.425943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 74.437867] alg: No test for adler32 (adler32-zlib) [ 75.834785] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 76.663964] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 78.423399] Key type lgssc registered [ 80.096641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.738512] hrtimer: interrupt took 7203199 ns [ 101.053450] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 145.840990] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 159.309523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 159.367894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.726994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 160.767574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 160.873301] Lustre: lustre-MDT0000: new disk, initializing [ 160.987894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 161.003643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.453386] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.921837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.014407] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 180.048195] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 180.056374] Lustre: Skipped 1 previous similar message [ 180.135431] Lustre: lustre-MDT0001: new disk, initializing [ 180.211730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.241420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 180.261764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.687518] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.265515] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 198.867451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.167731] Lustre: lustre-OST0000: new disk, initializing [ 199.171809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.185447] Lustre: 8457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.266481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.837758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 203.849183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 203.945921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.807186] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.834193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.928348] Lustre: lustre-OST0001: new disk, initializing [ 219.933050] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 219.939454] Lustre: 9530:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.007942] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.951889] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.429622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 229.442889] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 229.521595] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 238.920595] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 248.583242] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 255.189904] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing check_logdir /tmp/testlogs/ [ 260.917829] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing yml_node [ 266.130555] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 268.766395] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 271.743262] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 273.755449] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:35:04 EDT 2026 [ 292.750379] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 294.218995] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 311.266103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.268441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.281192] Lustre: Skipped 1 previous similar message [ 311.301913] Lustre: Skipped 3 previous similar messages [ 316.402710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 316.424550] Lustre: Skipped 3 previous similar messages [ 316.729438] Lustre: server umount lustre-MDT0000 complete [ 320.741798] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438953 with bad export cookie 10368974285044307726 [ 320.748703] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.754315] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 321.515049] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.515713] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.534462] Lustre: Skipped 3 previous similar messages [ 321.551000] Lustre: Skipped 1 previous similar message [ 325.542987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.553016] Lustre: Skipped 1 previous similar message [ 330.658814] LustreError: 6528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.660758] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.680149] LustreError: 6528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 330.696697] Lustre: Skipped 1 previous similar message [ 335.327160] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 335.586736] Lustre: server umount lustre-MDT0001 complete [ 339.900183] Lustre: server umount lustre-OST0000 complete [ 343.852257] Lustre: server umount lustre-OST0001 complete [ 350.938630] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 360.634885] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 399.916892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.288903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 400.356507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 400.446436] Lustre: lustre-MDT0000: new disk, initializing [ 400.525634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.538710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 405.192701] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 416.194536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.354141] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 416.432496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 416.438996] Lustre: Skipped 1 previous similar message [ 416.541714] Lustre: lustre-MDT0001: new disk, initializing [ 416.691386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 416.725848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 416.738716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 421.413412] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 431.892425] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 438.662198] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 445.971438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 446.254839] Lustre: lustre-OST0000: new disk, initializing [ 446.258526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 446.268952] Lustre: 17057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 446.378633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.657298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 447.676830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 447.839941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 452.942398] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 463.283753] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 467.290165] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 467.536546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 472.152892] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 472.475941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 478.697087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.697726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 478.740303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 478.745020] LustreError: Skipped 1 previous similar message [ 478.755982] Lustre: Skipped 1 previous similar message [ 480.938314] Lustre: server umount lustre-OST0000 complete [ 488.421411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 488.438698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.459696] Lustre: Skipped 1 previous similar message [ 493.837550] Lustre: server umount lustre-MDT0000 complete [ 494.049536] LustreError: 15969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.067390] LustreError: 15969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 497.619709] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439130 with bad export cookie 10368974285044309259 [ 497.623187] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 497.631182] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 497.979760] Lustre: server umount lustre-MDT0001 complete [ 502.897519] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:38:53 (1788439133) [ 505.367357] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 507.743462] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:38:58 (1788439138) [ 516.162815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.696229] LustreError: 19466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.773682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 521.715322] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 522.211168] LustreError: 19467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.312325] LustreError: 19466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.856773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.499876] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 541.654178] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 547.019339] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 554.294555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 561.192119] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 569.743703] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 577.505830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 577.513490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.515342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 577.515350] Lustre: Skipped 2 previous similar messages [ 577.561361] Lustre: Skipped 2 previous similar messages [ 583.688385] Lustre: server umount lustre-OST0000 complete [ 587.750315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.368558] Lustre: server umount lustre-MDT0000 complete [ 597.697506] LustreError: 19447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439230 with bad export cookie 10368974285044310134 [ 597.701163] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.710865] LustreError: 19447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 597.996775] Lustre: server umount lustre-MDT0001 complete [ 610.660956] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 614.567291] Key type lgssc unregistered [ 614.947138] LNet: 22996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.964950] LNetError: 22996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.994244] LNet: Removed LNI 192.168.203.131@tcp [ 616.355276] Key type .llcrypt unregistered [ 616.357396] Key type ._llcrypt unregistered [ 646.117547] Key type ._llcrypt registered [ 646.121126] Key type .llcrypt registered [ 646.217502] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 647.382479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 647.566747] alg: No test for adler32 (adler32-zlib) [ 648.827588] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 649.211433] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 651.039178] Key type lgssc registered [ 652.596951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 661.753522] loop: module loaded [ 661.852516] loop0: detected capacity change from 0 to 204800000 [ 662.146510] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 662.168863] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 662.594475] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 669.548079] loop0: detected capacity change from 0 to 204800000 [ 669.727841] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 669.745857] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 669.760721] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 675.475680] loop0: detected capacity change from 0 to 204800000 [ 681.894119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 681.962193] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 683.399764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 683.501880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 683.648901] Lustre: lustre-MDT0000: new disk, initializing [ 683.837739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 683.887488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 688.817696] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 698.983903] loop1: detected capacity change from 0 to 204800000 [ 705.267086] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 705.760648] Lustre: lustre-OST07c6: new disk, initializing [ 705.766031] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 705.771630] Lustre: Skipped 1 previous similar message [ 705.777985] Lustre: 26181:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 705.912265] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 711.817132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 711.821488] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 711.863111] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 712.989151] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 717.793707] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 717.806373] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 717.811586] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 717.815790] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 717.820213] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 717.828587] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 735.358165] Lustre: 26697:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 735.374144] Lustre: 26697:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 735.387426] Lustre: 26697:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 735.398259] Lustre: 26697:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 735.407538] Lustre: 26697:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 735.414389] Lustre: 26697:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 740.502293] Lustre: 26697:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 740.518271] Lustre: 26697:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 740.530089] Lustre: 26697:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 740.542599] Lustre: 26697:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.550727] Lustre: 26697:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 740.565052] Lustre: 26697:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.574389] Lustre: 26697:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 740.583224] Lustre: 26697:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.594439] Lustre: 26697:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 740.605740] Lustre: 26697:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.613217] Lustre: 26697:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 740.626213] Lustre: 26697:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.813561] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 743.837578] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 743.852889] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 743.866712] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.878761] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 743.886533] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.896520] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 743.910145] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.921270] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 743.935731] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.945297] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 743.957635] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.792769] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 748.799862] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 748.807869] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 748.812159] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.820640] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 748.829686] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.837420] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 748.847598] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.859684] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 748.869582] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.876943] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 748.885330] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.963103] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 756.976862] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 756.983617] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 756.991496] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 756.997763] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 757.021405] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 757.028919] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 757.039448] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 757.059426] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 757.063610] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 757.068453] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 757.077749] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 808.929530] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 808.956480] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 813.864325] Lustre: server umount lustre-OST07c6 complete [ 823.554924] Lustre: server umount lustre-MDT0000 complete [ 843.054597] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 846.308968] Key type lgssc unregistered [ 846.705982] LNet: 28736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.716532] LNetError: 28736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.783077] LNet: Removed LNI 192.168.203.131@tcp [ 849.021101] Key type .llcrypt unregistered [ 849.024799] Key type ._llcrypt unregistered [ 874.373862] Key type ._llcrypt registered [ 874.375416] Key type .llcrypt registered [ 874.515549] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 875.765609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 875.967053] alg: No test for adler32 (adler32-zlib) [ 877.054853] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 877.279522] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 878.887171] Key type lgssc registered [ 880.354463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 889.358443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 889.406666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 891.088560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 895.931431] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 900.582950] LustreError: 30003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.593243] LustreError: 30003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 905.214899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.638108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 911.333171] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 919.214373] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 925.297169] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 932.236097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 932.580728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.672636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 938.692228] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 947.017607] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 971.748294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.756852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 971.774178] Lustre: Skipped 1 previous similar message [ 971.802058] Lustre: Skipped 1 previous similar message [ 974.001239] Lustre: server umount lustre-OST0000 complete [ 981.985108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.987863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 982.004953] Lustre: Skipped 1 previous similar message [ 982.034798] Lustre: Skipped 1 previous similar message [ 984.362152] Lustre: server umount lustre-MDT0000 complete [ 987.104542] LustreError: 30003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 987.130836] LustreError: 30003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 988.394828] LustreError: 29982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439621 with bad export cookie 10985599189177119313 [ 988.397279] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 988.401781] LustreError: 29982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 988.856842] Lustre: server umount lustre-MDT0001 complete [ 1001.679650] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 1005.094842] Key type lgssc unregistered [ 1005.370851] LNet: 33740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.388346] LNetError: 33740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.421297] LNet: Removed LNI 192.168.203.131@tcp [ 1006.435412] Key type .llcrypt unregistered [ 1006.439524] Key type ._llcrypt unregistered [ 1025.595504] Key type ._llcrypt registered [ 1025.600426] Key type .llcrypt registered [ 1025.692674] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 1026.826647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1026.879114] alg: No test for adler32 (adler32-zlib) [ 1027.871212] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1028.148186] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 1029.847794] Key type lgssc registered [ 1031.136705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1039.473258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1039.509325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1041.069732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.644793] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1050.593560] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1050.626699] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1054.810828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.472235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1060.539430] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1068.452509] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1075.264495] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1082.824842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1083.128518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1086.195107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1089.058992] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1097.121245] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1112.034092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1112.037673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.050248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1114.085546] Lustre: server umount lustre-OST0000 complete [ 1121.251513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.264867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1121.275033] Lustre: Skipped 1 previous similar message [ 1121.286149] Lustre: Skipped 1 previous similar message [ 1125.674793] Lustre: server umount lustre-MDT0000 complete [ 1126.368392] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1126.420297] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1129.488311] LustreError: 34987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439762 with bad export cookie 11931074164903102571 [ 1129.489855] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1129.497663] LustreError: 34987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1139.875756] Lustre: server umount lustre-MDT0001 complete [ 1152.060461] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 1154.693960] Key type lgssc unregistered [ 1155.028647] LNet: 38564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.038975] LNetError: 38564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.063600] LNet: Removed LNI 192.168.203.131@tcp [ 1155.833760] Key type .llcrypt unregistered [ 1155.835860] Key type ._llcrypt unregistered [ 1184.473488] Key type ._llcrypt registered [ 1184.476932] Key type .llcrypt registered [ 1184.589936] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 1185.702757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1185.858195] alg: No test for adler32 (adler32-zlib) [ 1187.038432] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1187.337112] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 1189.042564] Key type lgssc registered [ 1190.340846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1198.286143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1198.304463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1199.778497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1203.366845] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1209.314111] LustreError: 39979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.337680] LustreError: 39979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1213.284701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.827908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1219.225615] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1229.224268] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1237.044537] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1245.232333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1245.616210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.650846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1252.286208] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1260.171542] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1267.639081] Lustre: server umount lustre-MDT0000 complete [ 1268.139097] LustreError: 41178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.227100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.255375] LustreError: 39972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.256902] Lustre: Skipped 1 previous similar message [ 1269.288206] LustreError: 39972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1271.564043] LustreError: 39960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439904 with bad export cookie 14831462736840961860 [ 1271.574197] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1273.221615] LustreError: 41178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.258224] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.31@tcp (stopping) [ 1274.341023] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.367635] Lustre: Skipped 1 previous similar message [ 1274.374552] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1276.388304] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1277.985703] Lustre: server umount lustre-MDT0001 complete [ 1307.103151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1307.313521] Lustre: server umount lustre-OST0000 complete [ 1324.554471] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 1327.894846] Key type lgssc unregistered [ 1328.230632] LNet: 43839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1328.236166] LNetError: 43839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.269432] LNet: Removed LNI 192.168.203.131@tcp [ 1329.645492] Key type .llcrypt unregistered [ 1329.652457] Key type ._llcrypt unregistered [ 1362.578864] Key type ._llcrypt registered [ 1362.580559] Key type .llcrypt registered [ 1362.702268] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 1364.100403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1364.131085] alg: No test for adler32 (adler32-zlib) [ 1365.324547] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1365.664754] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 1367.399191] Key type lgssc registered [ 1368.846379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1377.997122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1378.028206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.587927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1384.348768] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1390.054601] LustreError: 45253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.075628] LustreError: 45253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1393.157620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1393.690723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1398.165938] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1407.400930] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1414.345621] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1421.056623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1421.350854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1424.438247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1428.057942] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1436.101190] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1448.934944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.936127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1448.936514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1448.948792] Lustre: Skipped 1 previous similar message [ 1448.971027] Lustre: Skipped 1 previous similar message [ 1452.561316] Lustre: server umount lustre-OST0000 complete [ 1473.508282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1473.514434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1473.523273] Lustre: Skipped 1 previous similar message [ 1473.538404] Lustre: Skipped 2 previous similar messages [ 1478.626185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1483.744870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1483.751712] Lustre: Skipped 2 previous similar messages [ 1485.279168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1485.558223] Lustre: server umount lustre-MDT0000 complete [ 1488.874405] LustreError: 45249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.892982] LustreError: 45249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1489.478855] LustreError: 45988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440122 with bad export cookie 13467861566071178588 [ 1489.486222] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1489.493690] LustreError: 45988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1495.857747] Lustre: server umount lustre-MDT0001 complete [ 1509.020559] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 1513.313497] Key type lgssc unregistered [ 1513.760620] LNet: 48964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.765126] LNetError: 48964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.780985] LNet: Removed LNI 192.168.203.131@tcp [ 1514.794061] Key type .llcrypt unregistered [ 1514.795776] Key type ._llcrypt unregistered [ 1543.232674] Key type ._llcrypt registered [ 1543.234922] Key type .llcrypt registered [ 1543.346612] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 1544.858519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1544.870637] alg: No test for adler32 (adler32-zlib) [ 1545.865780] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1546.163478] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 1547.840521] Key type lgssc registered [ 1548.954360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1557.142513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1557.161220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1558.767250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1563.518461] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1568.230619] LustreError: 50384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.253145] LustreError: 50384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1572.455309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.959056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1577.742994] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1587.378640] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1593.118667] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1600.219236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1600.508546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.585939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1607.256619] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1616.171236] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1623.180815] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1627.157557] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1639.617620] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 8h57m50s [ 1641.544640] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1643.500573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.521379] Lustre: Skipped 1 previous similar message [ 1643.523883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1643.553366] Lustre: Skipped 2 previous similar messages [ 1644.455618] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.31@tcp (stopping) [ 1644.478420] Lustre: Skipped 1 previous similar message [ 1645.505448] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.31@tcp (stopping) [ 1645.516265] Lustre: Skipped 3 previous similar messages [ 1649.988904] Lustre: server umount lustre-MDT0000 complete [ 1650.570220] LustreError: 50380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1653.748646] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1656.707450] LustreError: 50379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.725482] LustreError: 50379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1661.259754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.474651] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1661.803053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1661.868300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1662.827689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1666.907754] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1670.872519] Lustre: DEBUG MARKER: Wait for df (36687) ... [ 1672.519976] Lustre: DEBUG MARKER: done [ 1677.282658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1677.292063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1677.319900] Lustre: Skipped 1 previous similar message [ 1677.329483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1678.311896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.425517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1683.436647] Lustre: Skipped 3 previous similar messages [ 1691.106140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1691.500657] Lustre: server umount lustre-OST0000 complete [ 1698.785727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1698.792207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.800282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1698.817529] Lustre: Skipped 1 previous similar message [ 1698.831338] Lustre: Skipped 3 previous similar messages [ 1702.560476] Lustre: server umount lustre-MDT0000 complete [ 1705.965289] LustreError: 50384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1705.984561] LustreError: 50384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1706.346165] LustreError: 53012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440339 with bad export cookie 3018361492746682756 [ 1706.359495] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1706.364024] LustreError: 53012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1706.673246] Lustre: server umount lustre-MDT0001 complete [ 1718.872387] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 1721.637880] Key type lgssc unregistered [ 1721.922165] LNet: 55311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.934487] LNetError: 55311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1721.955972] LNet: Removed LNI 192.168.203.131@tcp [ 1722.975779] Key type .llcrypt unregistered [ 1722.978163] Key type ._llcrypt unregistered [ 1765.616925] Key type ._llcrypt registered [ 1765.620682] Key type .llcrypt registered [ 1765.748347] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 1767.388925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1767.399652] alg: No test for adler32 (adler32-zlib) [ 1768.552943] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1768.826237] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 1770.503656] Key type lgssc registered [ 1771.890877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1779.899687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1779.948241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1781.334939] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1781.355867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1781.489919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.679632] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1794.247866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.447952] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1794.489201] Lustre: 57520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1794.800243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1799.844938] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1807.885352] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1813.834287] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1821.526866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1821.696176] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1821.904609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.801439] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 1829.537308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1837.572237] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1844.584503] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 9h01m15s [ 1846.273938] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1854.048834] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1856.974289] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1857.089173] LustreError: 57528:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1877.412580] Lustre: lustre-MDT0000: Client 3e2f822a-3088-4c51-b4a9-f122e073db78 (at 192.168.203.31@tcp) reconnecting [ 1897.191154] LustreError: 57528:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1900.073669] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1902.992326] Lustre: DEBUG MARKER: done [ 1911.264597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1911.270966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.303923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.777786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1911.797495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.816918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1915.930102] Lustre: server umount lustre-OST0000 complete [ 1922.025316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.036985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1922.055136] Lustre: Skipped 1 previous similar message [ 1925.978900] Lustre: server umount lustre-MDT0000 complete [ 1929.830722] LustreError: 57513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440562 with bad export cookie 3770077482777140893 [ 1929.833496] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1929.849862] LustreError: 57513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1930.168788] Lustre: server umount lustre-MDT0001 complete [ 1941.260892] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 1944.380723] Key type lgssc unregistered [ 1944.632632] LNet: 61858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.644167] LNetError: 61858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.666283] LNet: Removed LNI 192.168.203.131@tcp [ 1945.600339] Key type .llcrypt unregistered [ 1945.602834] Key type ._llcrypt unregistered [ 1991.735287] Key type ._llcrypt registered [ 1991.738460] Key type .llcrypt registered [ 1991.886565] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 1992.939459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.163528] alg: No test for adler32 (adler32-zlib) [ 1994.235604] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1994.438997] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 1996.111239] Key type lgssc registered [ 1997.624594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2001.670817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2001.689907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.906684] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2002.934581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2003.074130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.118685] Lustre: Failing over lustre-MDT0000 [ 2004.609290] Lustre: server umount lustre-MDT0000 complete [ 2012.528689] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:04:03 (1788440643) [ 2020.707172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.252690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.984167] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2036.793991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.919537] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2036.982877] Lustre: 64652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2037.106973] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2037.434868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2042.324593] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2051.237907] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2057.508908] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2064.987979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2065.269094] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2065.460505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.214022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2072.412537] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2080.081994] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2085.766380] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2093.030892] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.035679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2093.055575] Lustre: Skipped 2 previous similar messages [ 2093.073115] Lustre: Skipped 1 previous similar message [ 2098.146097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.154463] Lustre: Skipped 3 previous similar messages [ 2103.265379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.274239] Lustre: Skipped 2 previous similar messages [ 2106.336478] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2106.660518] Lustre: server umount lustre-MDT0000 complete [ 2111.021851] LustreError: 64645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440743 with bad export cookie 2293773583879892835 [ 2111.033190] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.039903] LustreError: 64645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2111.569922] Lustre: server umount lustre-MDT0001 complete [ 2115.537281] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2120.500485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2128.918179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2129.887773] Lustre: 63575:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440746/real 1788440746] req@000000003fc84002 x1875315918940160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440762 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2129.913980] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.959184] Lustre: 63577:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440749/real 1788440749] req@00000000f9b79288 x1875315918940288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440765 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2135.007430] Lustre: 63576:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440751/real 1788440751] req@000000008190a9ed x1875315918940416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440767 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2136.101303] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1fd51eebee55f846 [ 2136.113089] Lustre: MGC192.168.203.131@tcp: Connection restored to 0@lo (at 0@lo) [ 2136.362123] LustreError: 66504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.375394] LustreError: 66504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2136.450319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.534098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2137.439204] Lustre: 63574:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440753/real 1788440753] req@00000000ac81429c x1875315918940672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2138.468725] LustreError: 66505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.473843] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2139.566708] LustreError: 66504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.595398] LustreError: 66504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2141.599381] Lustre: 63575:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440758/real 1788440758] req@00000000162d1b6c x1875315918940800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440774 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2141.766801] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2144.738095] LustreError: 66505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.783905] LustreError: 66505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2149.861795] LustreError: 68425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.895390] LustreError: 68425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2152.307672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.696823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2157.875937] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2158.065800] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2165.860464] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2171.881341] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2183.655275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2183.657631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.682697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2183.686695] Lustre: Skipped 2 previous similar messages [ 2185.593988] Lustre: server umount lustre-MDT0000 complete [ 2188.769925] LustreError: 68816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.808381] LustreError: 68816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2189.424691] LustreError: 64647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440822 with bad export cookie 2293773583879895110 [ 2189.442393] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2189.443606] LustreError: 64647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2189.971681] Lustre: server umount lustre-MDT0001 complete [ 2197.350367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2208.607388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.247130] Lustre: 63577:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440826/real 1788440826] req@00000000d7b5627a x1875315918968832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440842 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2209.276271] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2209.288113] Lustre: Skipped 2 previous similar messages [ 2214.431657] LustreError: 63573:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000062ae0930 x1875315918969856/t0(0) o250->MGC192.168.203.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2214.802980] LustreError: 69162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.870058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.927546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2219.963790] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2229.423915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.225911] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2234.858183] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2238.957695] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2241.851636] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2248.135672] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2255.330102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2255.337316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.348245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.153396] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2269.357611] Lustre: server umount lustre-MDT0000 complete [ 2269.667925] LustreError: 71111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.678915] LustreError: 71111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2272.690374] LustreError: 64646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440905 with bad export cookie 2293773583879896489 [ 2272.692434] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2272.698600] LustreError: 64646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2272.945254] Lustre: server umount lustre-MDT0001 complete [ 2279.437822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2283.280771] Lustre: DEBUG MARKER: files compared the same [ 2295.973624] Lustre: server umount lustre-OST0000 complete [ 2314.074852] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 2318.184908] Key type lgssc unregistered [ 2318.618073] LNet: 74386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.624787] LNetError: 74386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2318.647972] LNet: Removed LNI 192.168.203.131@tcp [ 2319.809562] Key type .llcrypt unregistered [ 2319.813235] Key type ._llcrypt unregistered [ 2346.030537] Key type ._llcrypt registered [ 2346.032606] Key type .llcrypt registered [ 2346.194449] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 2347.491884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2347.581633] alg: No test for adler32 (adler32-zlib) [ 2348.712973] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2348.923079] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 2350.631226] Key type lgssc registered [ 2352.044922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2360.047550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2360.075247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2361.541965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2366.278550] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug +malloc all [ 2372.070433] LustreError: 75804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.092233] LustreError: 75804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2374.813021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.100899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2379.419920] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug +malloc all [ 2386.830298] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2392.857930] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2399.997711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2400.339448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2406.517382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2406.645567] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug +malloc all [ 2415.022624] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2427.368527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2427.375784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2427.393765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.436938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.437427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.443866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.235641] Lustre: server umount lustre-OST0000 complete [ 2436.576319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2436.594594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.606040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2436.613322] Lustre: Skipped 1 previous similar message [ 2440.672620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.675657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2440.690789] Lustre: Skipped 1 previous similar message [ 2441.958869] Lustre: server umount lustre-MDT0000 complete [ 2445.799716] LustreError: 75803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.821527] LustreError: 75803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2446.076350] LustreError: 75784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441078 with bad export cookie 7720896616172640002 [ 2446.090591] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2446.552305] Lustre: server umount lustre-MDT0001 complete [ 2458.101867] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 2460.905348] Key type lgssc unregistered [ 2461.231947] LNet: 79334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2461.244155] LNetError: 79334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2461.274683] LNet: Removed LNI 192.168.203.131@tcp [ 2462.027361] Key type .llcrypt unregistered [ 2462.032382] Key type ._llcrypt unregistered [ 2484.511606] Key type ._llcrypt registered [ 2484.514629] Key type .llcrypt registered [ 2484.599361] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 2485.300592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2485.328987] alg: No test for adler32 (adler32-zlib) [ 2486.550489] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2486.746603] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 2488.448759] Key type lgssc registered [ 2490.763941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2502.802940] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2502.840958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2518.431631] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2523.552096] LustreError: 80724:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.131@tcp: failed processing log, type 4: rc = -110 [ 2550.177950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2556.848676] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2566.507213] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2576.758040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.964486] Lustre: *** cfs_fail_loc=706, val=0*** [ 2577.446468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2577.539185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2582.166316] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2582.499374] LustreError: 80750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.527859] LustreError: 80750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2584.550985] LustreError: 80749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.583749] LustreError: 80749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2589.670638] LustreError: 81811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.710425] LustreError: 81811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2591.893225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.378592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2598.276089] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2607.931285] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2615.359268] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2618.345883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2618.370504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.412759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2618.421996] Lustre: Skipped 1 previous similar message [ 2623.457199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2623.464381] Lustre: Skipped 1 previous similar message [ 2628.580358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2628.590021] Lustre: Skipped 1 previous similar message [ 2632.159235] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2632.359417] Lustre: server umount lustre-OST0000 complete [ 2638.305655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2638.317717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2638.332843] Lustre: Skipped 1 previous similar message [ 2638.341490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.512169] Lustre: server umount lustre-MDT0000 complete [ 2643.938133] LustreError: 81814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.952697] LustreError: 81814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2646.274814] LustreError: 80730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441279 with bad export cookie 13815332510694739345 [ 2646.283646] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2646.284671] LustreError: 80730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2646.749787] Lustre: server umount lustre-MDT0001 complete [ 2657.352167] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 2660.177680] Key type lgssc unregistered [ 2660.482283] LNet: 84340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2660.488461] LNetError: 84340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2661.541635] LNet: Removed LNI 192.168.203.131@tcp [ 2662.379253] Key type .llcrypt unregistered [ 2662.388121] Key type ._llcrypt unregistered [ 2686.975501] Key type ._llcrypt registered [ 2686.978536] Key type .llcrypt registered [ 2687.075680] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 2687.932937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2688.051588] alg: No test for adler32 (adler32-zlib) [ 2689.221512] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2689.470824] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 2691.135179] Key type lgssc registered [ 2692.484101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2700.019302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2700.052428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.966701] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2714.116638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.817751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2719.855528] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2720.224866] LustreError: 86438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.252251] LustreError: 86438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2724.326150] LustreError: 86441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.353000] LustreError: 86441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2729.189528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2729.441919] LustreError: 86438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2729.474270] LustreError: 86438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2729.613430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2735.073341] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.105877] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2735.942143] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2740.203178] LustreError: 86441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2740.231871] LustreError: 86441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2744.380339] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2744.392830] Lustre: MGS: Client 4e2c1f80-95ff-44fc-973f-892a04b63861 (at 0@lo) reconnecting [ 2744.416573] LustreError: 86452:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2744.778478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2744.865986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2748.984825] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2765.281509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.292067] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2765.309303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2765.322774] Lustre: Skipped 1 previous similar message [ 2769.604949] Lustre: server umount lustre-OST0000 complete [ 2775.522370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2775.530087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.538405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2775.540693] Lustre: Skipped 2 previous similar messages [ 2775.558807] Lustre: Skipped 1 previous similar message [ 2779.296149] Lustre: server umount lustre-MDT0000 complete [ 2780.648268] LustreError: 87981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.663142] LustreError: 87981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2782.903086] Lustre: server umount lustre-MDT0001 complete [ 2786.987670] Lustre: server umount MGS complete [ 2800.620288] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:17:11 (1788441431) [ 2830.929854] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 2839.812330] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 2877.909101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.114377] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2893.116669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.384656] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2893.435748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2893.500076] Lustre: lustre-MDT0001: new disk, initializing [ 2893.671114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2893.723380] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2893.732439] Lustre: 93621:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2893.749216] LustreError: 93621:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2899.063729] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2909.693635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2909.992695] Lustre: lustre-OST0000: new disk, initializing [ 2909.996898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2910.003496] Lustre: 94386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2910.066454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2916.962188] Lustre: 94679:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c904dcb5 x1875316647697152/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788441565 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2917.238405] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2927.433893] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2933.929443] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2933.941164] Lustre: MGS: Client 303523c1-7f41-4279-a2f6-871157a2112e (at 0@lo) reconnecting [ 2933.990500] Lustre: 92887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2934.019508] LustreError: 92887:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2934.032214] LustreError: 92887:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2934.054141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2934.133970] Lustre: lustre-MDT0000: new disk, initializing [ 2934.243169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.263615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2934.287057] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2935.328444] Lustre: 95456:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000015f13862 x1875316647703936/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788441584 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2938.934368] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 2940.452730] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2944.425962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2944.444102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2944.523343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2969.073660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.078199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2969.096293] Lustre: Skipped 1 previous similar message [ 2974.181458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2974.190674] Lustre: Skipped 2 previous similar messages [ 2974.928714] Lustre: server umount lustre-OST0000 complete [ 2979.298038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2979.300831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2979.324372] Lustre: Skipped 1 previous similar message [ 2979.349221] Lustre: Skipped 1 previous similar message [ 2985.010105] Lustre: server umount lustre-MDT0000 complete [ 3003.359815] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3003.773026] Lustre: server umount lustre-MDT0001 complete [ 3008.525839] Lustre: server umount MGS complete [ 3024.594922] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:20:55 (1788441655) [ 3043.761066] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 3046.839719] Key type lgssc unregistered [ 3047.133544] LNet: 97957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3047.139684] LNetError: 97957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3047.165390] LNet: Removed LNI 192.168.203.131@tcp [ 3048.139322] Key type .llcrypt unregistered [ 3048.142812] Key type ._llcrypt unregistered [ 3065.256446] Key type ._llcrypt registered [ 3065.258156] Key type .llcrypt registered [ 3065.393272] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 3066.382664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3066.434305] alg: No test for adler32 (adler32-zlib) [ 3067.460347] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3067.642672] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 3069.287154] Key type lgssc registered [ 3070.300383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3078.444617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3078.474563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.914455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3084.676250] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3089.403029] LustreError: 99176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3089.428586] LustreError: 99176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3094.509391] LustreError: 99177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.381954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3096.858444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3102.420271] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3110.723386] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3116.999811] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3124.801783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3125.227381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3131.930823] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3140.281853] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3149.874536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3160.036334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3160.040843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.052383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3160.069202] Lustre: Skipped 1 previous similar message [ 3160.092575] Lustre: Skipped 1 previous similar message [ 3164.843400] Lustre: server umount lustre-OST0000 complete [ 3170.273592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3170.290730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3170.297532] Lustre: Skipped 1 previous similar message [ 3175.396610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3175.408535] Lustre: Skipped 1 previous similar message [ 3176.271344] Lustre: server umount lustre-MDT0000 complete [ 3179.504618] LustreError: 99173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3179.556224] LustreError: 99173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3180.772478] LustreError: 99157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441813 with bad export cookie 10420706793547414470 [ 3180.779827] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3180.782572] LustreError: 99157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3181.283863] Lustre: server umount lustre-MDT0001 complete [ 3194.270945] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 3198.285182] Key type lgssc unregistered [ 3198.740167] LNet: 102742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3198.746667] LNetError: 102742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3198.768423] LNet: Removed LNI 192.168.203.131@tcp [ 3200.163620] Key type .llcrypt unregistered [ 3200.168984] Key type ._llcrypt unregistered [ 3218.778246] Key type ._llcrypt registered [ 3218.791218] Key type .llcrypt registered [ 3219.033992] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 3220.829247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3220.964875] alg: No test for adler32 (adler32-zlib) [ 3222.281821] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3222.602636] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 3224.319177] Key type lgssc registered [ 3225.732572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3233.898714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3233.908733] LustreError: 103825:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3233.926570] LustreError: 103828:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3233.932480] LustreError: 103825:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 3233.941925] LustreError: 103825:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3233.949741] LustreError: 103825:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3233.952778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.488660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3243.389508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3243.729182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3247.833542] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3254.316678] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing lsmod [ 3257.439950] LustreError: 105029:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3257.491112] LustreError: 105033:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3257.498058] LustreError: 105029:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4959 [ 3257.503876] LustreError: 105029:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3257.523614] LustreError: 105029:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3257.536538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3257.891578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3261.006570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3264.481932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3264.500243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.535840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3266.018067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3266.018371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.031767] Lustre: Skipped 1 previous similar message [ 3266.043521] Lustre: Skipped 1 previous similar message [ 3267.642541] Lustre: server umount lustre-MDT0000 complete [ 3269.160550] LustreError: 103865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.175649] LustreError: 103865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3271.558859] LustreError: 103842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441904 with bad export cookie 2331828012233170460 [ 3271.569890] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3273.314434] LustreError: 104297:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000066af687e x1875317206569984/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3273.485063] Lustre: server umount lustre-MDT0001 complete [ 3287.968414] Lustre: server umount lustre-OST0000 complete [ 3296.768403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.200783] LustreError: 106095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3297.286657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3301.903291] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3302.382264] LustreError: 106096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3307.488245] LustreError: 106095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3310.005330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.454474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3315.413707] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3322.909648] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3328.912814] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3336.205391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3336.643234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3342.775494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3343.208604] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3351.874981] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3363.303491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3363.320820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.347117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3363.354799] Lustre: Skipped 1 previous similar message [ 3366.370270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3366.390055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3366.407161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.510437] Lustre: server umount lustre-OST0000 complete [ 3376.613293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3376.629456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3376.634542] Lustre: Skipped 2 previous similar messages [ 3386.848901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3386.854453] Lustre: Skipped 3 previous similar messages [ 3388.383190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3388.600964] Lustre: server umount lustre-MDT0000 complete [ 3391.970099] LustreError: 107745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.006221] LustreError: 107745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3392.486983] LustreError: 106077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442025 with bad export cookie 2331828012233171391 [ 3392.495488] LustreError: 106077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3392.505142] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3392.782651] Lustre: server umount lustre-MDT0001 complete [ 3404.351556] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 3407.485371] Key type lgssc unregistered [ 3407.866857] LNet: 109629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3407.875907] LNetError: 109629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3407.895317] LNet: Removed LNI 192.168.203.131@tcp [ 3408.779453] Key type .llcrypt unregistered [ 3408.785364] Key type ._llcrypt unregistered [ 3433.877830] Key type ._llcrypt registered [ 3433.879917] Key type .llcrypt registered [ 3434.013244] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 3435.641642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3435.680448] alg: No test for adler32 (adler32-zlib) [ 3437.087934] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3437.397558] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 3439.183339] Key type lgssc registered [ 3440.439556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3448.761609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3448.796552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3450.332837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.508212] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3460.577930] LustreError: 111048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.604110] LustreError: 111048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3464.632911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3465.127074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3469.393510] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3476.823905] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3482.600589] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3490.113886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3490.681990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3495.823425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3497.276330] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3505.588791] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3514.056071] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3517.683713] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3525.600579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3525.611679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.617438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3525.620787] Lustre: Skipped 1 previous similar message [ 3525.633670] Lustre: Skipped 1 previous similar message [ 3527.930729] Lustre: server umount lustre-OST0000 complete [ 3531.749188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3531.781328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3531.809907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3535.842492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3535.850786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.927868] Lustre: server umount lustre-MDT0000 complete [ 3540.962259] LustreError: 111048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3540.982474] LustreError: 111048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3542.020250] LustreError: 111028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442174 with bad export cookie 9619797250018325008 [ 3542.021108] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3542.030611] LustreError: 111028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3542.575603] Lustre: server umount lustre-MDT0001 complete [ 3555.267152] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 3558.907474] Key type lgssc unregistered [ 3559.260661] LNet: 114689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3559.267394] LNetError: 114689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3559.284872] LNet: Removed LNI 192.168.203.131@tcp [ 3560.364096] Key type .llcrypt unregistered [ 3560.375280] Key type ._llcrypt unregistered [ 3581.400797] Key type ._llcrypt registered [ 3581.403753] Key type .llcrypt registered [ 3581.531383] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 3582.552559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3582.648972] alg: No test for adler32 (adler32-zlib) [ 3583.728360] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3583.964879] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 3585.671326] Key type lgssc registered [ 3586.800232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3595.716765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3595.761387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.299213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3602.564438] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3607.526251] LustreError: 115909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3607.543681] LustreError: 115909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3612.372073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3612.641251] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3612.785761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3618.189938] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3626.570268] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3632.965712] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3640.663438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3641.143410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3643.191061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3648.077550] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3656.073190] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3667.948204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3667.949302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3667.968408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3667.977712] Lustre: Skipped 1 previous similar message [ 3667.988935] Lustre: Skipped 1 previous similar message [ 3671.364092] Lustre: server umount lustre-OST0000 complete [ 3678.191762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.194067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3678.207647] Lustre: Skipped 1 previous similar message [ 3678.226452] Lustre: Skipped 1 previous similar message [ 3681.887753] Lustre: server umount lustre-MDT0000 complete [ 3683.300107] LustreError: 115908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3683.321896] LustreError: 115908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3685.897452] LustreError: 115889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442318 with bad export cookie 18085139256294441799 [ 3685.914219] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3685.928878] LustreError: 115889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3686.189907] Lustre: server umount lustre-MDT0001 complete [ 3698.943618] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 3702.926313] Key type lgssc unregistered [ 3703.142046] LNet: 119441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3703.152107] LNetError: 119441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3703.175255] LNet: Removed LNI 192.168.203.131@tcp [ 3703.858229] Key type .llcrypt unregistered [ 3703.864520] Key type ._llcrypt unregistered [ 3730.345728] Key type ._llcrypt registered [ 3730.350680] Key type .llcrypt registered [ 3730.486277] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 3731.589610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3731.696850] alg: No test for adler32 (adler32-zlib) [ 3732.799579] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3733.085893] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 3734.807504] Key type lgssc registered [ 3735.948666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3744.205248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3744.245106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3745.742347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3750.750306] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3756.004645] LustreError: 120855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3756.021198] LustreError: 120855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3759.557460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3760.048833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3764.354945] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3772.152831] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3777.758918] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3784.603380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3784.984032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3791.178826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3791.988240] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3800.025619] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3811.231930] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3815.172229] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3815.176983] Lustre: Skipped 2 previous similar messages [ 3817.087659] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3817.129929] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3819.834612] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3819.844735] Lustre: Skipped 2 previous similar messages [ 3825.251917] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3825.255417] Lustre: Skipped 1 previous similar message [ 3832.605383] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.31@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3832.611470] Lustre: Skipped 2 previous similar messages [ 3832.704993] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.31@tcp 192.168.0.[2,10]@tcp [ 3838.868887] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.31/24@tcp in log lustre-MDT0000 [ 3838.877666] Lustre: Skipped 2 previous similar messages [ 3840.599122] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.31/24@tcp [ 3840.607061] Lustre: Skipped 1 previous similar message [ 3850.724718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.726340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3850.738950] Lustre: Skipped 1 previous similar message [ 3850.771227] Lustre: Skipped 1 previous similar message [ 3854.092573] Lustre: server umount lustre-OST0000 complete [ 3860.962452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.991590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3861.002595] Lustre: Skipped 1 previous similar message [ 3864.881060] Lustre: server umount lustre-MDT0000 complete [ 3866.081573] LustreError: 120854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3866.096840] LustreError: 120854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3868.901472] LustreError: 120834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442501 with bad export cookie 8455058837067349392 [ 3868.911317] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3868.914973] LustreError: 120834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3869.266188] Lustre: server umount lustre-MDT0001 complete [ 3880.873375] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 3884.089887] Key type lgssc unregistered [ 3884.436327] LNet: 125602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3884.443465] LNetError: 125602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3884.466209] LNet: Removed LNI 192.168.203.131@tcp [ 3885.545589] Key type .llcrypt unregistered [ 3885.547670] Key type ._llcrypt unregistered [ 3916.702319] Key type ._llcrypt registered [ 3916.707277] Key type .llcrypt registered [ 3916.829774] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 3917.588947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3917.714504] alg: No test for adler32 (adler32-zlib) [ 3918.865624] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3919.146151] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 3920.872764] Key type lgssc registered [ 3922.025786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3930.970535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3930.990476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3932.470510] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3932.498566] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.31/24@tcp [ 3932.501174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3937.355191] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3942.885755] LustreError: 127163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3942.919621] LustreError: 127163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3946.122369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3946.423796] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3946.445426] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.31/24@tcp [ 3946.511388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3951.011815] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3958.557994] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3965.064898] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3972.386630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3972.746391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3977.934352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3980.380352] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 3990.549121] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4003.814963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4003.819497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4003.826511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.802135] Lustre: server umount lustre-OST0000 complete [ 4012.002301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4012.019618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4012.027521] Lustre: Skipped 1 previous similar message [ 4014.968413] Lustre: server umount lustre-MDT0000 complete [ 4017.126503] LustreError: 127163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4017.158815] LustreError: 127163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4019.181443] LustreError: 127142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442651 with bad export cookie 2701169640590888200 [ 4019.189331] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4019.200299] LustreError: 127142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4029.581315] Lustre: server umount lustre-MDT0001 complete [ 4041.140594] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 4044.631183] Key type lgssc unregistered [ 4045.082075] LNet: 130750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4045.092289] LNetError: 130750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4045.121830] LNet: Removed LNI 192.168.203.131@tcp [ 4046.314951] Key type .llcrypt unregistered [ 4046.318350] Key type ._llcrypt unregistered [ 4076.931486] Key type ._llcrypt registered [ 4076.937056] Key type .llcrypt registered [ 4077.068085] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 4092.761812] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 4094.042665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4094.133952] alg: No test for adler32 (adler32-zlib) [ 4095.193798] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4095.434732] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 4097.103193] Key type lgssc registered [ 4098.198314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4133.135827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4133.146694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.378176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4134.392286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4134.445559] Lustre: lustre-MDT0000: new disk, initializing [ 4134.508522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4134.524832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4137.993528] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4149.634299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4149.789501] Lustre: 134151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4149.878870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4149.882413] Lustre: Skipped 1 previous similar message [ 4149.989369] Lustre: lustre-MDT0001: new disk, initializing [ 4150.122191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4150.179074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4150.196737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4155.096087] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4165.051962] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4171.400852] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4178.810508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4179.173377] Lustre: lustre-OST0000: new disk, initializing [ 4179.177737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4179.190757] Lustre: 136119:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4179.339475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4186.228378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4186.242313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4186.334631] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4186.376739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4196.999227] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4202.152127] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4202.401375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4207.607376] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4207.845466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4212.198156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4212.202064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4212.206925] LustreError: Skipped 1 previous similar message [ 4212.234087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4212.240281] Lustre: Skipped 1 previous similar message [ 4215.951962] Lustre: server umount lustre-OST0000 complete [ 4221.928958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4221.936282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.944947] Lustre: Skipped 1 previous similar message [ 4221.949245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4227.553612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4227.566158] Lustre: Skipped 3 previous similar messages [ 4232.673165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4232.689601] Lustre: Skipped 1 previous similar message [ 4234.207256] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4234.408243] Lustre: server umount lustre-MDT0000 complete [ 4239.120749] LustreError: 134142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442871 with bad export cookie 6410009079541681055 [ 4239.124104] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4239.132298] LustreError: 134142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4239.572891] Lustre: server umount lustre-MDT0001 complete [ 4250.099148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.903535] LustreError: 138338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4250.923469] LustreError: 138338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4251.039104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4256.129565] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4256.230646] LustreError: 138338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4256.246940] LustreError: 138338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4260.338788] LustreError: 138339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.717841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.941232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4269.280388] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4276.740792] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4285.763740] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4295.906508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4296.666702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4304.143166] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4312.636340] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4318.401887] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4318.570796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4322.874364] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4323.018157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4331.949447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4332.228399] Lustre: lustre-OST0001: new disk, initializing [ 4332.237804] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4332.266073] Lustre: 141712:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4332.287612] Lustre: lustre-OST0001: Not available for connect from 192.168.203.31@tcp (not set up) [ 4332.414563] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4339.321217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4339.334177] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4339.461718] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4339.592954] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4349.352416] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4349.625918] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4354.919287] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4355.115149] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4361.290576] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f6c8386a000.ost_server_uuid 50 [ 4363.013159] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f6c8386a000.ost_server_uuid in IDLE state after 0 sec [ 4375.535829] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4375.538407] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4375.541932] LustreError: Skipped 1 previous similar message [ 4375.560406] Lustre: Skipped 1 previous similar message [ 4375.567223] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4375.587602] Lustre: Skipped 1 previous similar message [ 4377.906301] Lustre: server umount lustre-OST0001 complete [ 4380.654396] LustreError: 140163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4380.696954] LustreError: 140163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4384.737182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4384.750816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.770152] Lustre: Skipped 1 previous similar message [ 4385.768339] LustreError: 140162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4385.798572] LustreError: 140162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4389.421694] Lustre: server umount lustre-OST0000 complete [ 4396.001384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4396.016421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4396.023478] Lustre: Skipped 2 previous similar messages [ 4396.047186] Lustre: Skipped 4 previous similar messages [ 4399.825692] Lustre: server umount lustre-MDT0000 complete [ 4401.125146] LustreError: 139359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4401.152038] LustreError: 139359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4404.131156] LustreError: 138318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443036 with bad export cookie 6410009079541681937 [ 4404.135281] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4404.146620] LustreError: 138318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4404.903531] Lustre: server umount lustre-MDT0001 complete [ 4428.682673] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 4431.938384] Key type lgssc unregistered [ 4432.344327] LNet: 144716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4432.384521] LNetError: 144716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4432.409356] LNet: Removed LNI 192.168.203.131@tcp [ 4433.407622] Key type .llcrypt unregistered [ 4433.416339] Key type ._llcrypt unregistered [ 4476.661867] Key type ._llcrypt registered [ 4476.665398] Key type .llcrypt registered [ 4476.786335] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 4491.894481] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 4493.213107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4493.246714] alg: No test for adler32 (adler32-zlib) [ 4494.373623] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4494.653532] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 4496.335249] Key type lgssc registered [ 4497.406856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4538.522994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4538.544214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4539.973458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4540.086273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4540.297659] Lustre: lustre-MDT0000: new disk, initializing [ 4540.563472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4540.593080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4546.678208] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4558.622702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.689092] Lustre: 148873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4558.735768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4558.742692] Lustre: Skipped 1 previous similar message [ 4558.793424] Lustre: lustre-MDT0001: new disk, initializing [ 4558.872121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4558.904327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4558.914633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4562.618242] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4572.907963] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4578.808664] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4586.036379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4586.368883] Lustre: lustre-OST0000: new disk, initializing [ 4586.372633] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4586.380679] Lustre: 150840:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4586.493773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4593.462727] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4595.287016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4595.303630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4595.444232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4602.358210] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4606.356052] Lustre: Failing over lustre-OST0000 [ 4606.489620] Lustre: server umount lustre-OST0000 complete [ 4624.373561] Lustre: 147093:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443241/real 1788443241] req@0000000086f345ac x1875318540370688/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788443257 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4624.400223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.115218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4625.401399] Lustre: lustre-OST0000: Not available for connect from 192.168.203.31@tcp (not set up) [ 4625.469305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.500548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4625.526952] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4626.469816] Lustre: 147095:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443243/real 1788443243] req@00000000f79531d3 x1875318540371968/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788443259 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4626.495704] Lustre: 147095:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4629.542794] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4631.520232] Lustre: 147094:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443248/real 1788443248] req@0000000028a2aec3 x1875318540373120/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788443264 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4631.559596] Lustre: 147094:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4631.963366] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4634.705865] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4634.707349] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4635.862249] Lustre: Failing over lustre-MDT0000 [ 4636.113420] Lustre: server umount lustre-MDT0000 complete [ 4637.920230] Lustre: 147094:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443254/real 1788443254] req@00000000881eac79 x1875318540374144/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788443270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4637.961134] Lustre: 147094:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4639.713501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.743725] Lustre: Skipped 2 previous similar messages [ 4644.837462] LustreError: 148883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4644.867575] LustreError: 148883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4645.766217] LustreError: 148880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4649.954025] LustreError: 150849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4649.970887] LustreError: 150849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4654.394407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4654.661834] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4654.914145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.969435] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4656.013459] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4659.118426] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4660.210945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4660.221482] Lustre: Skipped 1 previous similar message [ 4660.243584] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4660.286475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4670.434314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4670.443329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4670.451425] Lustre: Skipped 1 previous similar message [ 4670.464198] Lustre: Skipped 1 previous similar message [ 4673.395341] Lustre: server umount lustre-OST0000 complete [ 4680.677425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4680.683925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.692745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4680.706197] Lustre: Skipped 1 previous similar message [ 4683.366985] Lustre: server umount lustre-MDT0000 complete [ 4685.793824] LustreError: 148884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.826968] LustreError: 148884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4687.326193] LustreError: 148863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443320 with bad export cookie 15370136171859344424 [ 4687.327728] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4687.347613] LustreError: 148863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4687.876282] Lustre: server umount lustre-MDT0001 complete [ 4699.334222] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 4702.047419] Key type lgssc unregistered [ 4702.428870] LNet: 154499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4702.437621] LNetError: 154499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4702.465692] LNet: Removed LNI 192.168.203.131@tcp [ 4703.384809] Key type .llcrypt unregistered [ 4703.388176] Key type ._llcrypt unregistered [ 4742.898709] Key type ._llcrypt registered [ 4742.903329] Key type .llcrypt registered [ 4743.048496] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 4744.072942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4744.102316] alg: No test for adler32 (adler32-zlib) [ 4745.155802] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4745.461921] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 4747.207157] Key type lgssc registered [ 4748.263438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4755.383794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4755.396568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4756.920196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4761.411489] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4767.204773] LustreError: 156018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4767.233168] LustreError: 156018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4771.233757] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4772.330050] LustreError: 156019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4775.191190] LustreError: 156047:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4775.203794] LustreError: 156047:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4775.211978] LustreError: 156047:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4775.458302] Lustre: server umount lustre-MDT0000 complete [ 4790.749746] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 4799.922736] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 4832.423267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.643832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4832.672475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4832.792375] Lustre: lustre-MDT0000: new disk, initializing [ 4832.903895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4832.916961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4837.665964] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4848.970042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4849.164966] Lustre: 159347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4849.242553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4849.251487] Lustre: Skipped 1 previous similar message [ 4849.316589] Lustre: lustre-MDT0001: new disk, initializing [ 4849.381451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4849.458009] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4849.475456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4853.368327] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4863.110751] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4867.959743] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4874.498723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4874.756758] Lustre: lustre-OST0000: new disk, initializing [ 4874.766235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4874.784652] Lustre: 161317:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4874.885813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4880.649711] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 4885.075991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4885.098782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4885.235575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4889.672686] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4894.500909] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4896.285829] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:52:07 (1788443527) [ 5187.551706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5187.552624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5187.581947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5192.681642] LustreError: 160567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5192.701778] LustreError: 160567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5192.835352] Lustre: server umount lustre-MDT0000 complete [ 5196.708137] LustreError: 159340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443829 with bad export cookie 17327883400625603968 [ 5196.710968] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5196.728904] LustreError: 159340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5197.795477] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.796787] LustreError: 162885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5197.816302] Lustre: Skipped 2 previous similar messages [ 5197.870832] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5197.884723] Lustre: Skipped 2 previous similar messages [ 5201.835794] LustreError: 159355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5201.837852] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5206.945887] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5211.103132] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5211.590804] Lustre: server umount lustre-MDT0001 complete [ 5222.379920] Lustre: server umount lustre-OST0000 complete [ 5233.152670] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 5243.190224] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 5282.452821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5282.752610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5282.797641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5282.901623] Lustre: lustre-MDT0000: new disk, initializing [ 5282.991264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5283.012443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5288.019743] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 5299.230392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5299.342452] Lustre: 165988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5299.381341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5299.384398] Lustre: Skipped 1 previous similar message [ 5299.460565] Lustre: lustre-MDT0001: new disk, initializing [ 5299.554290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5299.602902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5299.615863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5304.681394] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 5315.016420] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5321.195234] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5329.492657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5329.887384] Lustre: lustre-OST0000: new disk, initializing [ 5329.895702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5329.925195] Lustre: 167957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5330.086884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5336.190688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5336.226277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5336.482730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5339.069763] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 5350.647473] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5355.804186] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5356.098413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5361.062227] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5361.346889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5367.265739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5367.270234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.283063] LustreError: Skipped 1 previous similar message [ 5367.285335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5367.301144] Lustre: Skipped 1 previous similar message [ 5367.320020] Lustre: Skipped 1 previous similar message [ 5369.760945] Lustre: server umount lustre-OST0000 complete [ 5376.480925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5376.484240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5376.490753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5380.002889] Lustre: server umount lustre-MDT0000 complete [ 5382.629497] LustreError: 167397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5382.667147] LustreError: 167397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5385.676263] LustreError: 165981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444018 with bad export cookie 17327883400625762798 [ 5385.678769] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5385.687877] LustreError: 165981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5386.257985] Lustre: server umount lustre-MDT0001 complete [ 5403.288240] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:00:33 (1788444033) [ 5424.639516] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 5436.329559] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 5480.641822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.033570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5481.132625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5481.254748] Lustre: lustre-MDT0000: new disk, initializing [ 5481.355253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5481.371198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5486.796727] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 5502.491553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.624439] Lustre: 172728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5502.929728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5509.267861] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 5519.347804] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5526.679682] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5535.473163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5535.868732] Lustre: lustre-OST0000: new disk, initializing [ 5535.872080] Lustre: Skipped 1 previous similar message [ 5535.877183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5535.892173] Lustre: Skipped 2 previous similar messages [ 5535.908874] Lustre: 174709:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5542.140564] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 5542.485557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5542.494776] Lustre: Skipped 1 previous similar message [ 5542.504427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5542.666171] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5553.308636] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5563.369016] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5563.375078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5563.388789] LustreError: Skipped 1 previous similar message [ 5563.402833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5563.414864] Lustre: Skipped 2 previous similar messages [ 5563.441898] Lustre: Skipped 3 previous similar messages [ 5569.654584] Lustre: server umount lustre-OST0000 complete [ 5574.624433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5574.651115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.102727] Lustre: server umount lustre-MDT0000 complete [ 5583.843819] LustreError: 172739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.866967] LustreError: 172739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5584.509368] LustreError: 174708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444217 with bad export cookie 17327883400625763680 [ 5584.512042] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5584.517383] LustreError: 174708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5584.867642] Lustre: server umount lustre-MDT0001 complete [ 5594.880166] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:03:45 (1788444225) [ 5612.887793] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 5622.694749] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 5659.358902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5659.632173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5659.660100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5659.761664] Lustre: lustre-MDT0000: new disk, initializing [ 5659.866152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5659.875979] Lustre: Skipped 2 previous similar messages [ 5659.901911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5664.943251] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 5676.612329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5676.694274] Lustre: 179037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5676.959435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5683.255489] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 5693.961612] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5700.595641] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5707.921612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5708.213497] Lustre: 181011:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5715.528656] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 5716.614710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5716.791599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5727.179298] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5737.440473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5737.446878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.462938] LustreError: Skipped 1 previous similar message [ 5737.466053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5737.481514] Lustre: Skipped 2 previous similar messages [ 5737.485930] Lustre: Skipped 5 previous similar messages [ 5741.017050] Lustre: server umount lustre-OST0000 complete [ 5752.088435] Lustre: server umount lustre-MDT0000 complete [ 5752.803566] LustreError: 179047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5752.830905] LustreError: 179047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5756.888671] LustreError: 179029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444389 with bad export cookie 17327883400625764758 [ 5756.890352] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5756.905228] LustreError: 179029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5757.345429] Lustre: server umount lustre-MDT0001 complete [ 5770.482275] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 5774.073624] Key type lgssc unregistered [ 5774.304521] LNet: 182867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5774.322262] LNetError: 182867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5774.360980] LNet: Removed LNI 192.168.203.131@tcp [ 5775.341449] Key type .llcrypt unregistered [ 5775.343064] Key type ._llcrypt unregistered [ 5800.073484] Key type ._llcrypt registered [ 5800.075734] Key type .llcrypt registered [ 5800.183609] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 5800.976881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5801.017195] alg: No test for adler32 (adler32-zlib) [ 5802.270892] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5802.558020] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 5804.239674] Key type lgssc registered [ 5805.786586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5814.368705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5814.397403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.010488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5821.613767] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 5825.507651] LustreError: 184278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5825.531740] LustreError: 184278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5830.633365] LustreError: 184277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5831.534628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5832.022149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5837.741294] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 5846.346304] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5852.261325] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5861.293938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5861.965576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5868.431635] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 5876.469752] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5885.554090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5906.401282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5906.408459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5906.438415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5906.449623] Lustre: Skipped 1 previous similar message [ 5911.002876] Lustre: server umount lustre-OST0000 complete [ 5916.648675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.658916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5916.666375] Lustre: Skipped 2 previous similar messages [ 5916.681240] Lustre: Skipped 1 previous similar message [ 5921.303657] Lustre: server umount lustre-MDT0000 complete [ 5921.770070] LustreError: 184277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5921.796801] LustreError: 184277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5925.054567] LustreError: 184257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444557 with bad export cookie 9692105185736774120 [ 5925.060931] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5925.066493] LustreError: 184257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5925.397777] Lustre: server umount lustre-MDT0001 complete [ 5937.919158] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 5940.966757] Key type lgssc unregistered [ 5941.222833] LNet: 187838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5941.227405] LNetError: 187838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5941.236784] LNet: Removed LNI 192.168.203.131@tcp [ 5942.056045] Key type .llcrypt unregistered [ 5942.063568] Key type ._llcrypt unregistered [ 5971.274870] Key type ._llcrypt registered [ 5971.276665] Key type .llcrypt registered [ 5971.445346] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 5972.471866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5972.492563] alg: No test for adler32 (adler32-zlib) [ 5973.578805] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5973.838594] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 5975.535376] Key type lgssc registered [ 5977.105636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5985.994346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5986.043085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5987.574072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5992.418423] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 5998.064376] LustreError: 189251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5998.098536] LustreError: 189251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6001.822905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6002.242731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6007.496861] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6016.356570] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6023.324498] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6030.940656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6031.364654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6032.444254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6038.711338] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6046.862247] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6054.837433] Lustre: lustre-OST0000: Not available for connect from 192.168.203.31@tcp (stopping) [ 6056.949244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6056.952595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6056.967283] Lustre: Skipped 1 previous similar message [ 6056.990186] Lustre: Skipped 1 previous similar message [ 6059.910962] Lustre: lustre-OST0000: Not available for connect from 192.168.203.31@tcp (stopping) [ 6062.050177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6067.179159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6067.190261] Lustre: Skipped 2 previous similar messages [ 6068.191181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6068.390445] Lustre: server umount lustre-OST0000 complete [ 6076.363916] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8f6c9296e000.ost_server_uuid 50 [ 6078.401135] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6092.770650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6092.801846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6092.823782] Lustre: Skipped 1 previous similar message [ 6098.923090] LustreError: 189250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6098.952808] LustreError: 189250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6099.079229] Lustre: server umount lustre-MDT0000 complete [ 6103.250715] LustreError: 189232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444736 with bad export cookie 15796151682246106240 [ 6103.252721] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6103.254679] LustreError: 189232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6103.688404] Lustre: server umount lustre-MDT0001 complete [ 6115.032815] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:12:25 (1788444745) [ 6123.284994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6123.799378] LustreError: 193172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6123.869716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6129.000692] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6129.124965] LustreError: 193173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6133.216962] LustreError: 193172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6138.710704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6138.994601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6143.697586] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6152.403916] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6159.214779] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6167.387627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6167.780582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6168.859768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6175.315494] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6183.918592] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6191.132436] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6191.386772] Lustre: lustre-OST0001: new disk, initializing [ 6191.391372] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6191.401785] Lustre: 196013:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6191.488752] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6196.882391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6196.896982] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6197.029531] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6198.463473] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6208.535270] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6212.584829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6212.605280] Lustre: Skipped 2 previous similar messages [ 6212.613710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6212.630039] Lustre: Skipped 3 previous similar messages [ 6217.662390] Lustre: server umount lustre-OST0000 complete [ 6217.705205] LustreError: 194992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6217.723074] LustreError: 194992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6224.802696] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6225.061470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6230.569749] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6230.833807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6235.012192] LustreError: 196271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.029480] LustreError: 196271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6248.428904] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6248.437903] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6248.441168] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6248.441179] Lustre: Skipped 1 previous similar message [ 6248.446156] Lustre: Skipped 1 previous similar message [ 6251.117995] Lustre: server umount lustre-OST0001 complete [ 6256.608693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6256.627123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6261.846878] Lustre: server umount lustre-MDT0000 complete [ 6265.929942] LustreError: 193153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444898 with bad export cookie 15796151682246107311 [ 6265.933194] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6265.940553] LustreError: 193153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6266.283552] Lustre: server umount lustre-MDT0001 complete [ 6295.921980] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:15:26 (1788444926) [ 6304.946783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6305.374728] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6305.407644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6305.674221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6310.875224] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6321.422406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6321.670956] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6321.733980] Lustre: 199455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6327.880886] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6336.286807] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6343.992491] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6351.855471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6352.124797] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6359.220165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6360.250903] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6368.569137] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6375.474216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6375.579070] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6375.843557] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6375.850756] Lustre: Skipped 2 previous similar messages [ 6382.677145] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6384.905287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6391.295655] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6394.848858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6394.868896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6394.889943] Lustre: Skipped 1 previous similar message [ 6394.893885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6394.899849] Lustre: Skipped 4 previous similar messages [ 6400.829055] Lustre: server umount lustre-OST0000 complete [ 6402.441842] LustreError: 201304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6402.453493] LustreError: 201304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6423.248994] Lustre: server umount lustre-OST0001 complete [ 6429.670365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6429.683431] LustreError: Skipped 1 previous similar message [ 6429.696538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6429.733088] Lustre: Skipped 1 previous similar message [ 6437.401062] Lustre: server umount lustre-MDT0000 complete [ 6441.444951] Lustre: 188561:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445058/real 1788445058] req@00000000dbba2c4f x1875320091349504/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788445074 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6442.277139] LustreError: 199448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445075 with bad export cookie 15796151682246108718 [ 6442.291143] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6442.883515] Lustre: server umount lustre-MDT0001 complete [ 6467.727155] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:18:19 (1788445099) [ 6488.236340] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 6499.713994] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 6538.190331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6538.409388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6538.462804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6538.579212] Lustre: lustre-MDT0000: new disk, initializing [ 6538.651783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6538.678225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6544.308316] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6556.711208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6556.897622] Lustre: 207554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6557.028309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6557.035054] Lustre: Skipped 1 previous similar message [ 6557.235312] Lustre: lustre-MDT0001: new disk, initializing [ 6557.424035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6557.435911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6562.550700] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6573.380853] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6578.988882] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6586.415766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6586.664209] Lustre: lustre-OST0000: new disk, initializing [ 6586.668743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6586.679695] Lustre: 209525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6592.104340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6592.124907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6592.277196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6594.012820] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6604.714345] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6609.338691] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6609.547540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6614.647940] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6614.901837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6618.087402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6618.088374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6618.107366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6618.107375] Lustre: Skipped 10 previous similar messages [ 6618.138978] Lustre: Skipped 3 previous similar messages [ 6631.391124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6631.606837] Lustre: server umount lustre-OST0000 complete [ 6640.879827] Lustre: server umount lustre-MDT0000 complete [ 6643.680925] LustreError: 207566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6643.716687] LustreError: 207566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6645.655143] LustreError: 207548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445278 with bad export cookie 15796151682246110503 [ 6645.661841] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6645.669795] LustreError: 207548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6646.242657] Lustre: server umount lustre-MDT0001 complete [ 6655.785787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.374973] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6671.806721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6678.064211] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6686.573101] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6693.078578] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6700.188896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6708.395667] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6717.205650] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6721.842779] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6722.021587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6726.465413] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6726.675348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6732.771659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6732.788220] LustreError: Skipped 1 previous similar message [ 6732.791680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.807830] Lustre: Skipped 3 previous similar messages [ 6735.316431] Lustre: server umount lustre-OST0000 complete [ 6742.768368] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6748.153673] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6776.541651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6784.214867] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6793.753201] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6797.263801] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6797.480684] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6797.508668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6797.512722] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6797.570462] Lustre: Skipped 1 previous similar message [ 6802.920170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6802.936302] LustreError: Skipped 2 previous similar messages [ 6808.510788] Lustre: server umount lustre-OST0000 complete [ 6819.029763] Lustre: server umount lustre-MDT0000 complete [ 6823.029582] LustreError: 211723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445455 with bad export cookie 15796151682246111385 [ 6823.033153] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6823.043981] LustreError: 211723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6823.373682] Lustre: server umount lustre-MDT0001 complete [ 6833.312869] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:24:24 (1788445464) [ 6840.415254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.825697] LustreError: 217714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6840.853661] LustreError: 217714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6841.026928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6841.039941] Lustre: Skipped 6 previous similar messages [ 6847.397505] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6859.238137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.106619] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6874.125052] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6880.157626] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6887.626232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6893.825807] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6901.917711] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6907.127832] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6910.611293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6916.579664] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6916.931339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6924.735880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6924.889493] Lustre: lustre-OST0001: new disk, initializing [ 6924.892729] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6924.898883] Lustre: 221106:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6931.467882] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 6933.586788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6933.599116] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6933.710346] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6941.941762] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6948.158515] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6948.534933] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6954.264644] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6954.926759] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6959.607618] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.619893] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6959.656830] Lustre: Skipped 4 previous similar messages [ 6959.662686] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6959.662694] Lustre: Skipped 17 previous similar messages [ 6964.694429] Lustre: server umount lustre-OST0001 complete [ 6975.489348] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6975.854190] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6980.312045] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6980.489923] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7009.501571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7014.895863] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7014.908704] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7014.915182] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7016.100924] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7023.205929] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7024.811366] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7033.582266] Lustre: server umount lustre-OST0001 complete [ 7052.771260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7057.401681] Lustre: server umount lustre-OST0000 complete [ 7067.795349] Lustre: server umount lustre-MDT0000 complete [ 7072.140423] LustreError: 217694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445704 with bad export cookie 15796151682246112351 [ 7072.146317] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7072.157593] LustreError: 217694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7078.938301] Lustre: server umount lustre-MDT0001 complete [ 7108.261550] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:28:59 (1788445739) [ 7114.808175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7115.081443] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7115.115228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7120.117493] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7130.058680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.131487] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7130.150952] Lustre: 226352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7135.290761] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7142.987451] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7149.769391] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7158.335874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7158.448673] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7165.693508] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7174.485783] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7184.911842] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7185.047520] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7192.241396] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7200.455940] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7202.441368] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7207.508132] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7207.767347] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7213.570764] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7213.804704] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7219.899264] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f6c87fef800.ost_server_uuid 50 [ 7221.563099] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f6c87fef800.ost_server_uuid in IDLE state after 0 sec [ 7222.546111] Lustre: Permanently deactivating lustre-OST0001 [ 7222.572342] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7226.165418] Lustre: Permanently reactivating lustre-OST0001 [ 7231.459046] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7231.475381] LustreError: Skipped 2 previous similar messages [ 7231.483710] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7231.500449] Lustre: Skipped 7 previous similar messages [ 7237.196574] Lustre: server umount lustre-OST0001 complete [ 7241.710270] LustreError: 228199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7241.734110] LustreError: 228199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7247.614461] Lustre: server umount lustre-OST0000 complete [ 7257.720590] Lustre: server umount lustre-MDT0000 complete [ 7261.254123] LustreError: 228227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445894 with bad export cookie 15796151682246113639 [ 7261.256741] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7261.265305] LustreError: 228227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7261.617529] Lustre: server umount lustre-MDT0001 complete [ 7290.595721] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:32:01 (1788445921) [ 7299.528264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7299.951369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7300.001628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7300.012852] Lustre: Skipped 5 previous similar messages [ 7305.783493] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7314.923895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7315.085045] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7315.113370] Lustre: 233046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7319.440131] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7327.286669] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7333.433601] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7340.291297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7340.487315] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7340.501934] Lustre: Permanently deactivating lustre-OST0000 [ 7340.510453] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7346.533371] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7349.858791] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7355.134770] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7363.404910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7363.487139] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7363.731885] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7363.740930] Lustre: Skipped 11 previous similar messages [ 7371.326558] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7371.623262] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7380.955564] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7383.514938] Lustre: Permanently reactivating lustre-OST0000 [ 7383.525977] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7383.529395] Lustre: Skipped 2 previous similar messages [ 7390.339934] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 00000000a58876e5, cur 1788446023 deadline 1788446012 last 1788445982 [ 7390.343044] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7390.374454] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7390.402455] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7390.417264] Lustre: Skipped 1 previous similar message [ 7390.543278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7403.226895] Lustre: server umount lustre-OST0001 complete [ 7413.876934] Lustre: server umount lustre-OST0000 complete [ 7424.420606] Lustre: server umount lustre-MDT0000 complete [ 7428.333651] LustreError: 233797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446061 with bad export cookie 15796151682246115277 [ 7428.339539] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7428.341801] LustreError: 233797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7428.695071] Lustre: server umount lustre-MDT0001 complete [ 7442.418632] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 7445.348726] Key type lgssc unregistered [ 7445.585723] LNet: 237928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7445.595971] LNetError: 237928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7445.620605] LNet: Removed LNI 192.168.203.131@tcp [ 7446.343326] Key type .llcrypt unregistered [ 7446.346990] Key type ._llcrypt unregistered [ 7473.519343] Key type ._llcrypt registered [ 7473.521771] Key type .llcrypt registered [ 7473.624456] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 7474.802400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7474.825571] alg: No test for adler32 (adler32-zlib) [ 7476.160844] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7476.580243] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 7478.335213] Key type lgssc registered [ 7479.823974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7490.613070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7490.646264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.233366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7497.068196] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7501.803213] LustreError: 239404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7501.816718] LustreError: 239404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7502.829984] LustreError: 239403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7505.548420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.679658] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7505.718073] Lustre: 239391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7505.726520] Lustre: Permanently deactivating lustre-MDT0001 [ 7505.748377] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7506.057544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7508.169557] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7508.176521] LustreError: 239434:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7508.203539] LustreError: 239434:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7508.217396] LustreError: 239434:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7511.036326] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7519.568854] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7525.488351] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7534.319631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7534.708054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7536.775315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7537.780955] LustreError: 241244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7541.617221] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7543.279228] LustreError: 241245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7548.385286] LustreError: 241245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7551.065766] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7560.685689] LustreError: 241871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7560.692131] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7560.707369] LustreError: 241871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7561.112158] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7566.341573] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7566.352431] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7568.776415] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7577.780091] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7582.106151] Lustre: Permanently reactivating lustre-MDT0001 [ 7582.119305] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7582.123975] Lustre: Skipped 2 previous similar messages [ 7588.273845] Lustre: lustre-MDT0001: Client 78507f81-f9d0-4e91-8a8f-0d4955a0cce6 (at 192.168.203.31@tcp) reconnecting [ 7595.580445] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7597.427107] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7603.597130] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7603.815687] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7609.469821] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7609.705894] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7613.193541] Lustre: Permanently deactivating lustre-MDT0001 [ 7613.224296] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7613.231835] Lustre: Skipped 2 previous similar messages [ 7621.203418] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7632.866915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7632.873979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7632.889311] Lustre: Skipped 2 previous similar messages [ 7632.896536] Lustre: Skipped 1 previous similar message [ 7637.877133] Lustre: server umount lustre-MDT0000 complete [ 7637.987771] LustreError: 240568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7638.021696] LustreError: 240568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7641.953971] LustreError: 241264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446274 with bad export cookie 9394983026856953714 [ 7641.967609] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7641.973910] LustreError: 241264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7642.407519] Lustre: server umount lustre-MDT0001 complete [ 7648.603811] Lustre: server umount lustre-OST0000 complete [ 7660.462484] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:38:11 (1788446291) [ 7671.292659] Lustre: server umount lustre-OST0001 complete [ 7678.421625] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 7687.065864] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 7722.702167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7722.920459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7722.931487] Lustre: Skipped 2 previous similar messages [ 7722.960918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7723.030260] Lustre: lustre-MDT0000: new disk, initializing [ 7723.124454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7723.149807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7727.517618] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7737.315964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7737.446227] Lustre: 247708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7737.528631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7737.533588] Lustre: Skipped 1 previous similar message [ 7737.637970] Lustre: lustre-MDT0001: new disk, initializing [ 7737.761335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7737.806777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7737.814437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7742.707630] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7754.304562] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7761.581801] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7768.917297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7769.171321] Lustre: lustre-OST0000: new disk, initializing [ 7769.174894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7769.184982] Lustre: 249682:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7769.258975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7774.712460] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7777.372649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7777.387036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7777.563887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7783.736559] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7785.650085] LustreError: 248576:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7793.802811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7793.959185] Lustre: lustre-OST0001: new disk, initializing [ 7793.962349] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7793.967488] Lustre: 250795:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7795.744469] LustreError: 248576:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7795.766956] LustreError: 247718:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7800.789374] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7802.438069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7802.452590] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7802.553551] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7805.840121] LustreError: 247718:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7810.334508] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7812.149371] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7815.601183] Lustre: lustre-OST0001: Not available for connect from 192.168.203.31@tcp (stopping) [ 7815.606322] Lustre: Skipped 2 previous similar messages [ 7818.219372] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7818.227508] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7818.228543] Lustre: Skipped 1 previous similar message [ 7818.231655] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7818.254194] LustreError: Skipped 1 previous similar message [ 7820.555696] Lustre: server umount lustre-OST0001 complete [ 7820.679125] LustreError: 249663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7838.720681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7838.743636] Lustre: Skipped 1 previous similar message [ 7838.754806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7838.760890] Lustre: Skipped 2 previous similar messages [ 7843.809833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7843.822130] Lustre: Skipped 1 previous similar message [ 7851.999293] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7852.189529] Lustre: server umount lustre-OST0000 complete [ 7855.586491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7855.597052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7855.620311] Lustre: Skipped 1 previous similar message [ 7855.638091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7855.644256] Lustre: Skipped 2 previous similar messages [ 7861.906055] Lustre: server umount lustre-MDT0000 complete [ 7865.701555] LustreError: 247703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446498 with bad export cookie 9394983026856956318 [ 7865.703021] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7865.708732] LustreError: 247703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7880.159520] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7880.538729] Lustre: server umount lustre-MDT0001 complete [ 7890.982442] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 7893.463343] Key type lgssc unregistered [ 7893.874779] LNet: 252830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7893.880800] LNetError: 252830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7894.950681] LNet: Removed LNI 192.168.203.131@tcp [ 7895.741367] Key type .llcrypt unregistered [ 7895.743517] Key type ._llcrypt unregistered [ 7933.567925] Key type ._llcrypt registered [ 7933.570428] Key type .llcrypt registered [ 7933.746020] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 7934.688019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7934.783447] alg: No test for adler32 (adler32-zlib) [ 7935.860808] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7936.150524] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 7937.841950] Key type lgssc registered [ 7939.085253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7947.267846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7947.299645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7948.635518] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7948.666429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7948.827337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7953.110740] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7960.783912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7960.910887] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7960.950182] Lustre: 255034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7961.193529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7965.657128] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7974.027267] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7980.509397] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7987.073606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7987.274335] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7987.468857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7993.480492] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 7996.033597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8000.818582] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8052.196871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8052.209086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8052.221125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8052.713741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8052.728673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8052.758809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8057.623879] Lustre: server umount lustre-OST0000 complete [ 8063.039132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8078.562704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8079.243148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8085.935811] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8093.561740] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8093.575375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8093.576045] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8093.597310] Lustre: Skipped 1 previous similar message [ 8093.697807] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8134.625233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8134.630950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8134.638251] LustreError: Skipped 1 previous similar message [ 8134.644668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8134.649059] Lustre: Skipped 1 previous similar message [ 8134.665670] Lustre: Skipped 1 previous similar message [ 8138.227633] Lustre: server umount lustre-OST0000 complete [ 8144.870466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8144.887962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8147.752557] Lustre: server umount lustre-MDT0000 complete [ 8150.755915] LustreError: 255027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446783 with bad export cookie 3974399371082337948 [ 8150.759135] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8150.772296] LustreError: 255027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8151.030684] Lustre: server umount lustre-MDT0001 complete [ 8159.915172] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 8162.376167] Key type lgssc unregistered [ 8162.688562] LNet: 260452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8162.702336] LNetError: 260452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8162.733893] LNet: Removed LNI 192.168.203.131@tcp [ 8163.461826] Key type .llcrypt unregistered [ 8163.463422] Key type ._llcrypt unregistered [ 8185.661381] Key type ._llcrypt registered [ 8185.665391] Key type .llcrypt registered [ 8185.781640] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 8186.626858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8186.672640] alg: No test for adler32 (adler32-zlib) [ 8187.760926] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8187.968948] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 8189.615167] Key type lgssc registered [ 8191.120885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8199.707351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8199.718823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8201.160774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8205.706754] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8210.402548] LustreError: 261881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8210.423758] LustreError: 261881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8215.127647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8215.531139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8220.207755] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8228.497837] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8234.056467] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8240.703724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8241.019093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8246.141564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8247.168145] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8256.082808] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8282.080841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8282.092300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8282.097968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8282.119198] Lustre: Skipped 1 previous similar message [ 8282.143970] Lustre: Skipped 1 previous similar message [ 8287.145416] Lustre: server umount lustre-OST0000 complete [ 8292.321513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8292.323923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8292.341435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8292.351315] Lustre: Skipped 1 previous similar message [ 8297.442078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8297.461974] Lustre: Skipped 1 previous similar message [ 8302.562319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8305.120746] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8305.296037] Lustre: server umount lustre-MDT0000 complete [ 8307.682115] LustreError: 262610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8307.698888] LustreError: 262610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8309.092071] LustreError: 261861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446941 with bad export cookie 5804474502456207401 [ 8309.100562] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8309.105178] LustreError: 261861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8309.436957] Lustre: server umount lustre-MDT0001 complete [ 8319.241031] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 8321.686936] Key type lgssc unregistered [ 8321.945698] LNet: 266189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8321.957243] LNetError: 266189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8321.984854] LNet: Removed LNI 192.168.203.131@tcp [ 8322.960424] Key type .llcrypt unregistered [ 8322.963094] Key type ._llcrypt unregistered [ 8340.542234] Key type ._llcrypt registered [ 8340.543536] Key type .llcrypt registered [ 8340.783415] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 8342.436428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8342.596497] alg: No test for adler32 (adler32-zlib) [ 8343.948551] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8344.233596] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 8345.927368] Key type lgssc registered [ 8346.846720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8354.163612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8354.192831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8355.612291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8360.270571] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8366.052413] LustreError: 267405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8366.095450] LustreError: 267405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8370.040097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8370.555782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8374.227366] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8381.369910] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8386.792086] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8393.451390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8393.743907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8399.552354] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8400.951896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8406.557144] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8419.522586] Lustre: server umount lustre-OST0000 complete [ 8426.466821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8426.470394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8426.489141] Lustre: Skipped 1 previous similar message [ 8426.505067] Lustre: Skipped 1 previous similar message [ 8431.586826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8431.599164] Lustre: Skipped 1 previous similar message [ 8432.702920] Lustre: server umount lustre-MDT0000 complete [ 8436.365220] LustreError: 267385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447069 with bad export cookie 8609394106431655454 [ 8436.376485] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8436.384379] LustreError: 267385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8447.085498] Lustre: server umount lustre-MDT0001 complete [ 8456.605215] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 8459.284992] Key type lgssc unregistered [ 8459.599640] LNet: 271083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8459.618810] LNetError: 271083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8459.644211] LNet: Removed LNI 192.168.203.131@tcp [ 8460.486595] Key type .llcrypt unregistered [ 8460.489987] Key type ._llcrypt unregistered [ 8485.095269] Key type ._llcrypt registered [ 8485.098416] Key type .llcrypt registered [ 8485.212887] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 8486.087708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8486.219035] alg: No test for adler32 (adler32-zlib) [ 8487.308892] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8487.517397] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 8489.215301] Key type lgssc registered [ 8490.204420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8497.595161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8497.618544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8499.073375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8503.224655] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8508.389551] LustreError: 272499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8508.401275] LustreError: 272499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8511.773309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8512.176957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8515.685780] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8523.144120] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8529.601984] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8538.091392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8538.596049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8543.728308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8544.179625] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8551.807493] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8582.626325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8582.626331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8582.629626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8582.640290] Lustre: Skipped 1 previous similar message [ 8586.813384] Lustre: server umount lustre-OST0000 complete [ 8592.866771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8592.870021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8592.892543] Lustre: Skipped 1 previous similar message [ 8592.903655] Lustre: Skipped 2 previous similar messages [ 8597.919446] Lustre: server umount lustre-MDT0000 complete [ 8597.985820] LustreError: 272500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8598.002334] LustreError: 272500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8601.355242] LustreError: 272479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447234 with bad export cookie 5744645437057925906 [ 8601.360250] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8601.370503] LustreError: 272479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8601.613333] Lustre: server umount lustre-MDT0001 complete [ 8612.932296] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 8615.735845] Key type lgssc unregistered [ 8616.021538] LNet: 276805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8616.029049] LNetError: 276805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8616.042055] LNet: Removed LNI 192.168.203.131@tcp [ 8616.861680] Key type .llcrypt unregistered [ 8616.864350] Key type ._llcrypt unregistered [ 8634.546553] Key type ._llcrypt registered [ 8634.549813] Key type .llcrypt registered [ 8634.687844] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 8636.035969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8636.087781] alg: No test for adler32 (adler32-zlib) [ 8637.250292] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8637.544306] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 8639.215544] Key type lgssc registered [ 8640.567325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8648.211426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8648.234669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8649.782144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8654.669994] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8659.940371] LustreError: 278037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8659.956591] LustreError: 278037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8662.916779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8663.226537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8667.921531] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8674.779871] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8680.194181] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8686.768995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8687.096931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8693.486981] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8694.256242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8700.618520] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8713.696733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8713.706053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8713.740199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8714.727910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8714.729779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8714.767989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8717.533240] Lustre: server umount lustre-OST0000 complete [ 8724.960692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8724.971397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8725.003355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8725.036193] Lustre: Skipped 2 previous similar messages [ 8727.507655] Lustre: server umount lustre-MDT0000 complete [ 8730.083731] LustreError: 278040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8730.119981] LustreError: 278040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8732.730263] LustreError: 278004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447365 with bad export cookie 352456255283608757 [ 8732.742065] LustreError: 278004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8732.751420] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8733.227994] Lustre: server umount lustre-MDT0001 complete [ 8744.685807] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 8747.109949] Key type lgssc unregistered [ 8747.420212] LNet: 281740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8747.429840] LNetError: 281740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8747.445771] LNet: Removed LNI 192.168.203.131@tcp [ 8748.333315] Key type .llcrypt unregistered [ 8748.335543] Key type ._llcrypt unregistered [ 8759.416865] Key type ._llcrypt registered [ 8759.421351] Key type .llcrypt registered [ 8759.552769] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8778.237823] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 8792.632228] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 8793.777240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8793.832321] alg: No test for adler32 (adler32-zlib) [ 8794.884195] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8795.111096] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 8796.783178] Key type lgssc registered [ 8797.912026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8837.094819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8837.134148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.317217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8838.362320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8838.492362] Lustre: lustre-MDT0000: new disk, initializing [ 8838.609194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8838.651232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8842.897884] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8852.266861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8852.350436] Lustre: 285290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8852.397235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8852.401321] Lustre: Skipped 1 previous similar message [ 8852.458230] Lustre: lustre-MDT0001: new disk, initializing [ 8852.560926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8852.591760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8852.597555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8856.806428] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8866.539770] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8872.763140] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8880.403874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8880.668514] Lustre: lustre-OST0000: new disk, initializing [ 8880.671385] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8880.680884] Lustre: 287259:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8880.759766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8887.748269] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8890.987285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8890.993507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8891.171610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8897.151712] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8901.412914] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8901.553363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8905.681698] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8905.890602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8911.840295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8911.844050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8911.850894] LustreError: Skipped 1 previous similar message [ 8911.855449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8911.863017] Lustre: Skipped 1 previous similar message [ 8914.017369] Lustre: server umount lustre-OST0000 complete [ 8919.008780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8919.015252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8919.034251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8919.041667] Lustre: Skipped 1 previous similar message [ 8922.080051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8922.083726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8923.820639] Lustre: server umount lustre-MDT0000 complete [ 8927.464707] LustreError: 285285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447560 with bad export cookie 9367338092535828089 [ 8927.466482] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8927.484739] LustreError: 285285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8927.988200] Lustre: server umount lustre-MDT0001 complete [ 8938.792587] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:59:29 (1788447569) [ 8946.462342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.793125] LustreError: 289670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8946.809536] LustreError: 289670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8946.858650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8950.896417] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8952.290474] LustreError: 289670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8956.395485] LustreError: 289671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8959.175916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8959.550888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8963.728991] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8971.177446] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8977.137676] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8983.283809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8983.592166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8988.879965] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 8996.545828] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9005.027251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9005.035291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9005.049618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9005.064583] Lustre: Skipped 1 previous similar message [ 9011.053417] Lustre: server umount lustre-OST0000 complete [ 9015.776435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9015.780981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9015.798338] Lustre: Skipped 1 previous similar message [ 9015.805271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9015.810783] Lustre: Skipped 2 previous similar messages [ 9027.556450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9027.563633] Lustre: Skipped 5 previous similar messages [ 9029.088131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9029.200846] Lustre: server umount lustre-MDT0000 complete [ 9032.451453] LustreError: 289652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447665 with bad export cookie 9367338092535828971 [ 9032.461023] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9032.679236] Lustre: server umount lustre-MDT0001 complete [ 9042.460909] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 9045.224930] Key type lgssc unregistered [ 9045.568695] LNet: 293217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9045.576936] LNetError: 293217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9045.595537] LNet: Removed LNI 192.168.203.131@tcp [ 9046.333443] Key type .llcrypt unregistered [ 9046.339755] Key type ._llcrypt unregistered [ 9075.368338] Key type ._llcrypt registered [ 9075.375829] Key type .llcrypt registered [ 9075.484602] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 9091.221861] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 9092.217660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9092.291390] alg: No test for adler32 (adler32-zlib) [ 9093.521448] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9093.870050] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 9095.591157] Key type lgssc registered [ 9096.632587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9140.754462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9140.767320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9142.001866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9142.070259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9142.221909] Lustre: lustre-MDT0000: new disk, initializing [ 9142.388239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9142.420580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9147.062740] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9158.305583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9158.486067] Lustre: 297039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9158.574310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9158.581940] Lustre: Skipped 1 previous similar message [ 9158.685718] Lustre: lustre-MDT0001: new disk, initializing [ 9158.813916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9158.878182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9158.896327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9164.153625] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9173.197200] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9177.624987] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9183.560493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9183.808926] Lustre: lustre-OST2710: new disk, initializing [ 9183.812549] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9183.817701] Lustre: 299008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9183.916945] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9189.429822] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9190.457498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9190.479707] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9190.580497] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9197.294736] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9204.771908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9204.899673] Lustre: lustre-OST03e8: new disk, initializing [ 9204.906816] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9204.921587] Lustre: 300071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9205.052335] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9211.910303] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9213.612883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9213.626685] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9213.792570] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9220.617278] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9227.967136] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9228.175583] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9232.993782] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9233.248883] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9238.007542] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9238.323109] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9243.167820] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9243.442721] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9249.771323] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9249.792162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9249.798565] Lustre: Skipped 3 previous similar messages [ 9249.813276] Lustre: Skipped 3 previous similar messages [ 9250.683663] Lustre: server umount lustre-MDT0000 complete [ 9254.299534] LustreError: 297030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447887 with bad export cookie 3080624296251150000 [ 9254.309180] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9254.315334] LustreError: 297030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9254.821667] Lustre: server umount lustre-MDT0001 complete [ 9268.693119] Lustre: server umount lustre-OST2710 complete [ 9270.239358] Lustre: 294837:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447887/real 1788447887] req@00000000cfcc9903 x1875323362805248/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788447903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9270.270398] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9271.990886] Lustre: server umount lustre-OST03e8 complete [ 9289.939843] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [ 9299.373356] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 9335.783752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9336.022479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9336.057881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9336.156190] Lustre: lustre-MDT0000: new disk, initializing [ 9336.278412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9336.320135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9341.033876] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9352.293301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9352.451397] Lustre: 305317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9352.539958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9352.552364] Lustre: Skipped 1 previous similar message [ 9352.637226] Lustre: lustre-MDT0001: new disk, initializing [ 9352.704439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9352.760642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9352.792387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9356.936938] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9370.145578] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9379.200291] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9386.797302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9387.057421] Lustre: lustre-OST0000: new disk, initializing [ 9387.061536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9387.067864] Lustre: 307289:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9387.184487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9393.606531] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9393.830038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9393.854041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9393.972586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9403.110524] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9408.037623] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9408.326255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9412.778221] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9412.951190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9414.627703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9414.637215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9414.645276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9414.659265] Lustre: Skipped 1 previous similar message [ 9419.749265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9419.757655] Lustre: Skipped 2 previous similar messages [ 9420.784284] Lustre: server umount lustre-OST0000 complete [ 9424.354147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9424.363267] LustreError: Skipped 1 previous similar message [ 9424.371829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9424.399869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9429.986785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9429.992189] Lustre: Skipped 2 previous similar messages [ 9432.261400] Lustre: server umount lustre-MDT0000 complete [ 9435.018576] LustreError: 305311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448067 with bad export cookie 3080624296251151680 [ 9435.024888] LustreError: 305311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9435.025189] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9435.254440] Lustre: server umount lustre-MDT0001 complete [ 9442.204952] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:07:53 (1788448073) [ 9443.552485] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9445.152113] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:07:56 (1788448076) [ 9448.491186] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 9474.011636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9474.481635] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9474.509438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9474.807708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9479.453306] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9490.384643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9490.549787] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9490.592148] Lustre: 311148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9496.316786] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9520.750931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9520.871200] Lustre: Denying initial registration attempt from nid 192.168.203.131@tcp, specified as failover [ 9520.876826] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9520.884413] LustreError: 312602:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9520.888975] LustreError: 312602:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9520.896623] LustreError: 312602:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9520.987141] Lustre: server umount lustre-OST0000 complete [ 9520.989122] LustreError: 312602:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9529.314530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9529.317770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9529.338811] Lustre: Skipped 2 previous similar messages [ 9529.351032] Lustre: Skipped 1 previous similar message [ 9542.111138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9542.491519] Lustre: server umount lustre-MDT0000 complete [ 9546.730027] LustreError: 311141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448179 with bad export cookie 3080624296251152562 [ 9546.754377] LustreError: 311141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9546.759297] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9547.518653] Lustre: server umount lustre-MDT0001 complete [ 9560.773476] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 9563.368770] Key type lgssc unregistered [ 9563.684586] LNet: 313725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9563.689936] LNetError: 313725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9563.707716] LNet: Removed LNI 192.168.203.131@tcp [ 9564.677503] Key type .llcrypt unregistered [ 9564.679443] Key type ._llcrypt unregistered [ 9579.239179] Key type ._llcrypt registered [ 9579.241162] Key type .llcrypt registered [ 9579.358308] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 9580.115341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9580.149436] alg: No test for adler32 (adler32-zlib) [ 9581.309549] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9581.571522] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 9583.303440] Key type lgssc registered [ 9584.194472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9633.021204] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 9642.374112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9642.407986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9643.679502] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9643.697562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9643.903843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9648.926803] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9657.396952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9657.576078] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9657.629947] Lustre: 316492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9657.914228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9661.640434] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9668.752400] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9675.185595] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9696.883353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9697.279852] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9697.564717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9703.846855] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9711.395576] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9715.186537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9715.193304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9715.218946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9715.227500] LustreError: Skipped 1 previous similar message [ 9720.500206] Lustre: server umount lustre-OST0000 complete [ 9725.410663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9725.413974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9725.441438] Lustre: Skipped 2 previous similar messages [ 9725.470568] Lustre: Skipped 2 previous similar messages [ 9730.529273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9730.533726] Lustre: Skipped 1 previous similar message [ 9735.648868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9735.655107] Lustre: Skipped 1 previous similar message [ 9739.231203] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9739.514567] Lustre: server umount lustre-MDT0000 complete [ 9740.769655] LustreError: 316497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9740.791996] LustreError: 316497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9743.564486] LustreError: 316484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448376 with bad export cookie 2286009684827271430 [ 9743.568563] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9743.575055] LustreError: 316484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9744.062634] Lustre: server umount lustre-MDT0001 complete [ 9756.069590] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 9758.736727] Key type lgssc unregistered [ 9759.093372] LNet: 320062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9759.099379] LNetError: 320062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9759.116729] LNet: Removed LNI 192.168.203.131@tcp [ 9759.853788] Key type .llcrypt unregistered [ 9759.855773] Key type ._llcrypt unregistered [ 9785.928566] Key type ._llcrypt registered [ 9785.934122] Key type .llcrypt registered [ 9786.061618] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [ 9787.315194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9787.436808] alg: No test for adler32 (adler32-zlib) [ 9788.535472] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9788.841975] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [ 9790.511230] Key type lgssc registered [ 9791.815204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9800.643233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9800.672052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9802.227623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9807.210851] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9812.457880] LustreError: 321474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9812.486753] LustreError: 321474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9816.541711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9816.990341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9822.008990] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9831.572805] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9838.581934] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9849.351741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9849.621387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9857.890977] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9866.234548] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9882.557061] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.31@tcp (stopping) [ 9882.596456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9882.617475] Lustre: Skipped 2 previous similar messages [ 9885.860571] Lustre: server umount lustre-MDT0000 complete [ 9887.618671] LustreError: 323295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9889.975515] LustreError: 321455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448522 with bad export cookie 7812803732969813030 [ 9889.990247] LustreError: 321455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9889.990632] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9892.744664] LustreError: 323295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9892.756119] LustreError: 323295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9892.779630] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.31@tcp (stopping) [ 9892.798427] Lustre: Skipped 3 previous similar messages [ 9892.834489] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9894.882155] LustreError: 321471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9894.883312] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9894.912298] Lustre: Skipped 1 previous similar message [ 9894.915783] LustreError: 321471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9896.581279] Lustre: server umount lustre-MDT0001 complete [ 9903.340952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9912.665697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9914.335916] LustreError: 320783:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006016355d x1875324091546240/t0(0) o250->MGC192.168.203.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9914.660767] LustreError: 323311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9914.727298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9914.790656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9919.164916] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9924.982978] LustreError: 325092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.985522] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9925.009993] LustreError: 325092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9927.628956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9927.977096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9932.623887] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [ 9933.301256] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9940.939696] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9943.026509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9949.335706] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9950.763443] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9958.888591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9958.891594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9958.924190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9958.933600] Lustre: Skipped 1 previous similar message [ 9962.914711] Lustre: server umount lustre-OST0000 complete [ 9968.032718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9968.062620] Lustre: Skipped 1 previous similar message [ 9968.078438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9968.085475] Lustre: Skipped 1 previous similar message [ 9973.700323] Lustre: server umount lustre-MDT0000 complete [ 9974.242942] LustreError: 325091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9974.255359] LustreError: 325091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9977.146977] LustreError: 321455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448609 with bad export cookie 7812803732969820877 [ 9977.149693] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9977.162836] LustreError: 321455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9977.456966] Lustre: server umount lustre-MDT0001 complete [ 9988.982672] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [ 9991.604341] Key type lgssc unregistered [ 9991.920770] LNet: 327765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9991.933642] LNetError: 327765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9991.960824] LNet: Removed LNI 192.168.203.131@tcp [ 9992.903142] Key type .llcrypt unregistered [ 9992.906691] Key type ._llcrypt unregistered [10018.308454] Key type ._llcrypt registered [10018.310757] Key type .llcrypt registered [10018.438626] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [10019.617687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10019.705980] alg: No test for adler32 (adler32-zlib) [10020.777460] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10021.080140] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [10022.775127] Key type lgssc registered [10023.854360] Lustre: Echo OBD driver; http://www.lustre.org/ [10031.386864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10031.409566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10032.800185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10037.825853] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10042.351461] LustreError: 329177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10042.373922] LustreError: 329177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10046.229982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10046.633745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10051.654853] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10058.663816] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10064.031683] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10068.450374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10068.485504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10071.586221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10071.595797] Lustre: Skipped 2 previous similar messages [10073.834823] Lustre: server umount lustre-MDT0000 complete [10076.640643] LustreError: 329905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10076.661715] LustreError: 329905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10077.673818] LustreError: 329157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448710 with bad export cookie 3444775277630465555 [10077.682439] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10077.685481] LustreError: 329157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10077.994270] Lustre: server umount lustre-MDT0001 complete [10092.036092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10092.339196] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10092.374130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10092.577875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10098.052914] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10108.359788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10108.518362] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10108.568481] Lustre: 331761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10108.951100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10114.716428] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10122.044497] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10127.792694] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10137.474980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10137.715681] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10137.726776] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10137.739641] LustreError: 333646:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10137.751779] LustreError: 333646:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10137.768024] LustreError: 333646:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10137.876235] Lustre: server umount lustre-OST0000 complete [10137.882653] LustreError: 333646:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10149.296740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10149.458640] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10149.638121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10154.952554] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10158.674373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10161.558589] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10168.454154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10168.540350] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10168.634486] Lustre: lustre-OST0001: new disk, initializing [10168.640104] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10168.647661] Lustre: 335189:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10168.734487] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10174.362979] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10177.619422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10177.639019] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10177.745403] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10180.658675] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10185.457296] Lustre: server umount lustre-OST0001 complete [10188.263286] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10188.273918] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10188.283210] LustreError: Skipped 1 previous similar message [10188.288534] LustreError: 334657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10188.295881] Lustre: Skipped 2 previous similar messages [10188.320188] LustreError: 334657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10189.282786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10189.283763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10189.291633] LustreError: Skipped 1 previous similar message [10189.292800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10189.320152] Lustre: Skipped 1 previous similar message [10193.378111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10193.378726] LustreError: 334113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10193.393093] Lustre: Skipped 2 previous similar messages [10195.384020] Lustre: server umount lustre-OST0000 complete [10201.067271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10201.076874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10201.098850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10204.899771] Lustre: server umount lustre-MDT0000 complete [10208.721107] LustreError: 331755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448841 with bad export cookie 3444775277630466108 [10208.741676] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10208.745773] LustreError: 331755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10209.107621] Lustre: server umount lustre-MDT0001 complete [10218.503873] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [10221.372687] Key type lgssc unregistered [10221.642169] LNet: 337147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10221.655072] LNetError: 337147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10221.669187] LNet: Removed LNI 192.168.203.131@tcp [10222.527881] Key type .llcrypt unregistered [10222.533685] Key type ._llcrypt unregistered [10289.929735] Key type ._llcrypt registered [10289.933825] Key type .llcrypt registered [10290.057316] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [10304.970242] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [10305.779803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10305.929042] alg: No test for adler32 (adler32-zlib) [10307.122096] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10307.387327] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [10309.047167] Key type lgssc registered [10310.248898] Lustre: Echo OBD driver; http://www.lustre.org/ [10345.326367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10345.353039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10346.778785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10346.839702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10346.962265] Lustre: lustre-MDT0000: new disk, initializing [10347.099071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10347.135095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10352.753831] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10364.173651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10364.307578] Lustre: 342073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10364.354943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10364.357369] Lustre: Skipped 1 previous similar message [10364.401845] Lustre: lustre-MDT0001: new disk, initializing [10364.517707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10364.603870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10364.613146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10370.171447] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10380.728581] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10386.531456] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10393.538737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10393.795558] Lustre: lustre-OST0000: new disk, initializing [10393.799222] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10393.805380] Lustre: 344042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10393.904199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10400.786722] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10403.916379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10403.922228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10404.042089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10411.488667] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10417.290455] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10417.616344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10422.421154] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10422.632707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10429.924314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10429.927902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10429.937270] Lustre: Skipped 1 previous similar message [10429.942108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10435.044518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10435.058915] Lustre: Skipped 2 previous similar messages [10439.136117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10439.550780] Lustre: server umount lustre-OST0000 complete [10445.281990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10445.311664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10450.401521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10450.416254] Lustre: Skipped 2 previous similar messages [10451.311984] Lustre: server umount lustre-MDT0000 complete [10456.277798] LustreError: 342066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449089 with bad export cookie 2376859129649184738 [10456.288921] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10456.302071] LustreError: 342066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10456.916971] Lustre: server umount lustre-MDT0001 complete [10466.614867] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:24:57 (1788449097) [10475.242885] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:25:06 (1788449106) [10484.482200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10484.830372] LustreError: 346799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10484.840680] LustreError: 346799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10484.896252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10489.017275] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10490.347893] LustreError: 346800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10494.509642] LustreError: 346799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10497.817550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10498.237738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10502.829558] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10509.828716] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10515.829617] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10522.779190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10523.135795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10529.846973] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10538.825769] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10547.173820] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10549.240366] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10551.103250] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10553.002968] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10554.539776] Lustre: Failing over lustre-MDT0000 [10554.851081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10554.866281] Lustre: Skipped 1 previous similar message [10554.869857] LustreError: 348041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10554.966338] Lustre: server umount lustre-MDT0000 complete [10559.468414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10559.485889] LustreError: Skipped 1 previous similar message [10559.489043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10559.505779] LustreError: 346800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10559.517943] LustreError: 346800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10570.208733] LustreError: 346799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10570.229210] LustreError: 346799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10574.150315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10574.273721] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10574.596892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10574.686537] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10577.807250] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10579.975136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10580.020927] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10580.097571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10580.353290] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10590.528233] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10592.313134] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10596.765131] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10599.068580] Lustre: Failing over lustre-MDT0000 [10599.371143] Lustre: server umount lustre-MDT0000 complete [10600.429144] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10600.434232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10600.458546] LustreError: 348041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10600.458561] LustreError: 348041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10616.811414] Lustre: 340301:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449233/real 1788449233] req@000000000abffcef x1875324635802240/t0(0) o400->MGC192.168.203.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788449249 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10616.848516] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10619.217218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10627.051775] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x20fc4cc907e24a7e [10627.072049] Lustre: MGC192.168.203.131@tcp: Connection restored to 0@lo (at 0@lo) [10627.085138] Lustre: Skipped 2 previous similar messages [10627.429945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10627.488775] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10629.516622] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10632.616235] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10632.682911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10632.713035] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10632.778628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10643.531399] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10645.798486] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10653.153673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10653.157381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10653.172039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10653.205642] Lustre: Skipped 4 previous similar messages [10657.866724] Lustre: server umount lustre-MDT0000 complete [10658.275824] LustreError: 346800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10658.303659] LustreError: 346800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [10661.600199] LustreError: 346781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449294 with bad export cookie 2376859129649187454 [10661.609320] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10661.616081] LustreError: 346781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10662.066360] Lustre: server umount lustre-MDT0001 complete [10676.015140] Lustre: server umount lustre-OST0000 complete [10691.606528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10692.216800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10696.976715] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10705.068601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10709.087131] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10715.934948] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10721.620430] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10727.522719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10734.000978] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10736.951082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10741.117969] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10746.356940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10746.377540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10746.391898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10746.404762] Lustre: Skipped 2 previous similar messages [10752.192194] Lustre: server umount lustre-OST0000 complete [10756.576624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10756.581550] LustreError: Skipped 1 previous similar message [10762.659225] Lustre: server umount lustre-MDT0000 complete [10762.726073] LustreError: 353142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10762.765916] LustreError: 353142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10766.759916] LustreError: 353124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449399 with bad export cookie 2376859129649188203 [10766.769895] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10766.772523] LustreError: 353124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10767.165378] Lustre: server umount lustre-MDT0001 complete [10779.807718] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [10783.199245] Key type lgssc unregistered [10783.462810] LNet: 356672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10783.468858] LNetError: 356672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10783.499962] LNet: Removed LNI 192.168.203.131@tcp [10784.630226] Key type .llcrypt unregistered [10784.635578] Key type ._llcrypt unregistered [10811.619925] Key type ._llcrypt registered [10811.624867] Key type .llcrypt registered [10811.749535] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [10812.816212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10813.024602] alg: No test for adler32 (adler32-zlib) [10814.258390] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10814.567841] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [10816.319249] Key type lgssc registered [10817.604996] Lustre: Echo OBD driver; http://www.lustre.org/ [10827.543630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10827.561515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10829.310970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10834.525172] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10839.522280] LustreError: 358186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10839.541471] LustreError: 358186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10843.004333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10843.419290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10848.681170] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10857.291547] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10863.475311] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10871.013050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10871.311650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10874.426416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10877.750708] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10885.507988] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10898.917671] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10898.929139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10898.941378] Lustre: Skipped 1 previous similar message [10900.688633] Lustre: server umount lustre-MDT0000 complete [10904.044776] LustreError: 360448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10904.062053] LustreError: 360448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10904.278634] LustreError: 358169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449537 with bad export cookie 7634033635133857590 [10904.281600] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10904.286748] LustreError: 358169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10904.591882] Lustre: server umount lustre-MDT0001 complete [10919.226415] Lustre: server umount lustre-OST0000 complete [10933.548394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10933.976983] LustreError: 361920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10934.055719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10938.982463] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10939.361198] LustreError: 361921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10939.372870] LustreError: 361921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10943.463788] LustreError: 361920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10947.476028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10947.764790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10952.249353] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10960.005653] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10965.854662] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10972.867038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10973.097165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10978.245139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10979.915223] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [10987.263964] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11002.853193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11002.856797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11002.866837] Lustre: Skipped 3 previous similar messages [11002.891927] Lustre: Skipped 4 previous similar messages [11006.668766] Lustre: server umount lustre-MDT0000 complete [11007.975867] LustreError: 363509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11007.999234] LustreError: 363509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11011.007965] LustreError: 361900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449643 with bad export cookie 7634033635133858703 [11011.024964] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11011.652196] Lustre: server umount lustre-MDT0001 complete [11025.437713] Lustre: server umount lustre-OST0000 complete [11040.557710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11040.898082] LustreError: 365833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11040.957520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11045.471401] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11055.266915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11061.836450] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11070.867946] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11077.055740] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11083.957889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11084.267106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11084.273297] Lustre: Skipped 1 previous similar message [11085.309292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11090.863704] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11099.168259] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11105.767931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11105.780540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11105.782603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11109.735535] Lustre: server umount lustre-OST0000 complete [11116.001561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11116.006484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11116.026143] Lustre: Skipped 2 previous similar messages [11116.029210] Lustre: Skipped 1 previous similar message [11121.121906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11121.138296] Lustre: Skipped 2 previous similar messages [11129.311190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11129.572284] Lustre: server umount lustre-MDT0000 complete [11130.342929] LustreError: 365834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11130.365099] LustreError: 365834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11133.457918] LustreError: 365813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449766 with bad export cookie 7634033635133859725 [11133.464759] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11133.469766] LustreError: 365813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11133.713671] Lustre: server umount lustre-MDT0001 complete [11145.403064] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [11148.425414] Key type lgssc unregistered [11148.727883] LNet: 369365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11148.749151] LNetError: 369365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11148.778579] LNet: Removed LNI 192.168.203.131@tcp [11149.782959] Key type .llcrypt unregistered [11149.788441] Key type ._llcrypt unregistered [11177.267450] Key type ._llcrypt registered [11177.269784] Key type .llcrypt registered [11177.384923] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [11178.377196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11178.571652] alg: No test for adler32 (adler32-zlib) [11179.719209] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11179.991605] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [11181.703169] Key type lgssc registered [11182.990641] Lustre: Echo OBD driver; http://www.lustre.org/ [11191.035383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11191.060506] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11191.070475] LustreError: 370801:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11191.084381] LustreError: 370801:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11191.090730] LustreError: 370801:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11191.097703] LustreError: 370801:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11191.103523] LustreError: 370801:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11199.689410] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11199.694285] LustreError: 371199:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11199.719306] LustreError: 371199:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11199.733150] LustreError: 371199:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11199.764869] LustreError: 371199:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11199.783138] LustreError: 371199:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11223.789280] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [11226.655651] Key type lgssc unregistered [11226.945938] LNet: 372202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11226.961700] LNetError: 372202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11226.979392] LNet: Removed LNI 192.168.203.131@tcp [11228.060337] Key type .llcrypt unregistered [11228.062873] Key type ._llcrypt unregistered [11246.920529] Key type ._llcrypt registered [11246.923961] Key type .llcrypt registered [11247.116280] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [11261.978745] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [11262.813498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11262.874872] alg: No test for adler32 (adler32-zlib) [11263.945357] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11264.229934] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [11265.959212] Key type lgssc registered [11267.456775] Lustre: Echo OBD driver; http://www.lustre.org/ [11307.914179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11307.958941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11309.246517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11309.288482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11309.404497] Lustre: lustre-MDT0000: new disk, initializing [11309.527474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11309.554464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11314.682266] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11325.774877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11325.990895] Lustre: 375356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11326.066793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11326.070510] Lustre: Skipped 1 previous similar message [11326.124802] Lustre: lustre-MDT0001: new disk, initializing [11326.230433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11326.261548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11326.277838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11331.017370] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11341.329420] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11348.363779] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11357.303648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11357.685324] Lustre: lustre-OST0000: new disk, initializing [11357.690637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11357.698309] Lustre: 377326:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11357.837763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11365.145584] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11368.071788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11368.086302] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11368.282660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11375.448772] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11380.354363] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11380.630895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11385.459645] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11385.684410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11388.899134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11388.902077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11388.916427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11388.920650] Lustre: Skipped 1 previous similar message [11394.020277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11394.036114] Lustre: Skipped 2 previous similar messages [11399.162762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11399.168034] Lustre: Skipped 1 previous similar message [11402.207358] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11402.484742] Lustre: server umount lustre-OST0000 complete [11408.352430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11408.364027] LustreError: Skipped 1 previous similar message [11408.370910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11408.391227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11409.380075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11412.668636] Lustre: server umount lustre-MDT0000 complete [11416.363451] LustreError: 375349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450049 with bad export cookie 5592214013254883260 [11416.371060] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11416.382280] LustreError: 375349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11416.746170] Lustre: server umount lustre-MDT0001 complete [11425.409525] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11427.186984] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:40:58 (1788450058) [11434.711091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11435.105634] LustreError: 379787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11435.122598] LustreError: 379787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11435.185067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11439.927962] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11440.613993] LustreError: 379788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11444.709929] LustreError: 379787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11449.847724] LustreError: 379788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11450.779733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11451.227451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11455.833842] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11464.629661] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11472.119840] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11480.727672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11481.098847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11487.602666] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11495.277533] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11502.362703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11502.515794] Lustre: lustre-OST0001: new disk, initializing [11502.519755] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11502.526222] Lustre: 382625:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11502.617728] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11509.813163] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11510.351501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11510.364286] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11510.466941] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11520.652633] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11526.113601] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11526.128395] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11526.128501] Lustre: Skipped 1 previous similar message [11526.138582] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11526.140066] Lustre: Skipped 2 previous similar messages [11530.770987] Lustre: server umount lustre-OST0001 complete [11531.237556] LustreError: 381612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11531.274589] LustreError: 381612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11541.474093] LustreError: 381612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11541.503486] LustreError: 381612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11550.688770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11550.696365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11550.718368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11550.727357] Lustre: Skipped 2 previous similar messages [11554.868231] Lustre: server umount lustre-OST0000 complete [11558.880766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11558.892290] LustreError: Skipped 1 previous similar message [11558.897571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11558.921550] Lustre: Skipped 1 previous similar message [11565.059680] Lustre: server umount lustre-MDT0000 complete [11567.078110] LustreError: 379788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11567.101950] LustreError: 379788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11568.997667] LustreError: 382624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450201 with bad export cookie 5592214013254884142 [11568.999291] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11569.006796] LustreError: 382624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11575.443980] Lustre: server umount lustre-MDT0001 complete [11585.223595] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [11588.150178] Key type lgssc unregistered [11588.414549] LNet: 384613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11588.431147] LNetError: 384613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11588.466407] LNet: Removed LNI 192.168.203.131@tcp [11589.387102] Key type .llcrypt unregistered [11589.390122] Key type ._llcrypt unregistered [11642.874774] Key type ._llcrypt registered [11642.878211] Key type .llcrypt registered [11643.027454] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [11644.417785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11644.456384] alg: No test for adler32 (adler32-zlib) [11645.663321] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11645.918204] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [11647.607148] Key type lgssc registered [11648.843813] Lustre: Echo OBD driver; http://www.lustre.org/ [11656.401498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11656.425729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11657.661385] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11657.691447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11657.865342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11662.134367] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11671.175981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11671.347335] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11671.387057] Lustre: 387406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11671.755664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11675.931705] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11683.683316] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11688.918203] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11692.002681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11692.009850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11692.026253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11692.514897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11692.531654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11692.534566] Lustre: Skipped 1 previous similar message [11697.635819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11697.641074] Lustre: Skipped 1 previous similar message [11702.755679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11702.763013] Lustre: Skipped 1 previous similar message [11705.311218] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11705.450252] Lustre: server umount lustre-MDT0000 complete [11708.856978] LustreError: 387398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450341 with bad export cookie 5668844916795375227 [11708.867386] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11708.870676] LustreError: 387398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11709.265478] Lustre: server umount lustre-MDT0001 complete [11718.673320] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:45:50 (1788450350) [11724.277611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11724.669329] LustreError: 389922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11724.682469] LustreError: 389922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11724.715617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11728.353381] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11729.893757] LustreError: 389923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11735.012282] LustreError: 389922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11736.388424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11736.686251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11741.149463] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11748.806678] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11754.588673] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11761.124774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11761.274747] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11761.463854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11767.393738] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11774.359642] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11782.204808] Lustre: Permanently deactivating lustre-OST0000 [11782.220383] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11789.931752] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11791.582888] LustreError: 392910:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11791.590501] LustreError: 392910:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11791.595876] LustreError: 392910:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11795.792690] LustreError: 392959:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11795.808453] LustreError: 392959:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11800.288647] Lustre: server umount lustre-OST0000 complete [11802.989746] LustreError: 393208:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11802.995634] LustreError: 393208:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11806.691208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11806.704451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11806.708334] Lustre: Skipped 1 previous similar message [11806.717552] Lustre: Skipped 1 previous similar message [11815.009848] Lustre: server umount lustre-MDT0000 complete [11816.930821] LustreError: 389923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11816.947580] LustreError: 389923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11818.519088] LustreError: 389903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450451 with bad export cookie 5668844916795375808 [11818.523534] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11818.529642] LustreError: 389903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11828.887450] Lustre: server umount lustre-MDT0001 complete [11837.347062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11842.231613] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11844.772930] LustreError: 394367:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11850.774313] Lustre: 394706:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.131@tcp [11852.867352] Lustre: server umount MGS complete [11863.893835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11864.300508] LustreError: 395430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.327122] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11864.332415] Lustre: Skipped 1 previous similar message [11864.397132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11868.890375] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11874.786959] LustreError: 395431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11874.814169] LustreError: 395431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11877.218578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11877.520072] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11882.178491] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11890.237542] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11895.811747] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11897.680079] Lustre: Permanently reactivating lustre-OST0000 [11897.698672] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11897.708520] Lustre: Skipped 2 previous similar messages [11925.985550] LustreError: 397236:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11933.870921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11934.447000] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11934.455561] Lustre: Skipped 1 previous similar message [11940.691532] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [11948.599252] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11956.000522] Lustre: server umount lustre-OST0000 complete [11959.788077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11959.793276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11959.803660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11959.806911] Lustre: Skipped 2 previous similar messages [11969.639956] Lustre: server umount lustre-MDT0000 complete [11971.553978] LustreError: 398487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11971.583270] LustreError: 398487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11972.576030] Lustre: 386729:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450589/real 1788450589] req@000000008457a185 x1875326038800000/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788450605 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11972.603515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11972.614191] Lustre: Skipped 1 previous similar message [11973.505428] LustreError: 395411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450606 with bad export cookie 5668844916795376858 [11973.508276] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11973.515271] LustreError: 395411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11973.820492] Lustre: server umount lustre-MDT0001 complete [11985.105965] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [11988.237285] Key type lgssc unregistered [11988.564506] LNet: 399894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11988.586491] LNetError: 399894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11988.625025] LNet: Removed LNI 192.168.203.131@tcp [11989.577947] Key type .llcrypt unregistered [11989.581125] Key type ._llcrypt unregistered [12010.870221] Key type ._llcrypt registered [12010.877807] Key type .llcrypt registered [12011.000302] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [12026.802326] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [12028.085833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12028.204138] alg: No test for adler32 (adler32-zlib) [12029.318236] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12029.581788] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [12031.319181] Key type lgssc registered [12032.549336] Lustre: Echo OBD driver; http://www.lustre.org/ [12070.008123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12070.033751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12071.387973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12071.434467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12071.531617] Lustre: lustre-MDT0000: new disk, initializing [12071.670571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12071.692614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12076.379216] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12088.353729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12088.598209] Lustre: 403048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12088.700251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12088.704506] Lustre: Skipped 1 previous similar message [12088.827463] Lustre: lustre-MDT0001: new disk, initializing [12088.977326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12089.018206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12089.024259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12094.624553] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12105.210145] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12111.995653] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12119.345440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12119.700842] Lustre: lustre-OST0000: new disk, initializing [12119.705241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12119.710729] Lustre: 405012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12119.794323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12125.798026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12125.817495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12125.959306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12127.086811] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12136.545805] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12141.876720] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12142.120467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12146.997224] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12147.245771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12151.777099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12151.786445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12151.792835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12155.388617] Lustre: server umount lustre-OST0000 complete [12160.992621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12160.997765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12161.006251] Lustre: Skipped 1 previous similar message [12161.011127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12161.014512] Lustre: Skipped 1 previous similar message [12162.020186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12162.022670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12167.140915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12167.153532] Lustre: Skipped 2 previous similar messages [12171.233815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12171.248692] Lustre: Skipped 1 previous similar message [12173.279338] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12173.456769] Lustre: server umount lustre-MDT0000 complete [12177.048381] LustreError: 403039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450809 with bad export cookie 4967209922336622491 [12177.056542] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12177.061645] LustreError: 403039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12177.325421] Lustre: server umount lustre-MDT0001 complete [12184.242629] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12185.911600] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:53:37 (1788450817) [12191.793489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12192.226626] LustreError: 407475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12192.246955] LustreError: 407475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12192.301371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12196.437351] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12197.345512] LustreError: 407476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12202.469893] LustreError: 407475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12204.456257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12204.691856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12208.122126] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12214.492530] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12219.371445] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12225.467638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12225.805065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12231.212790] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12237.877468] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12239.708294] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12242.047019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12248.547506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12248.553601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12248.558511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12248.561838] Lustre: Skipped 1 previous similar message [12248.583651] Lustre: Skipped 1 previous similar message [12253.839791] Lustre: server umount lustre-OST0000 complete [12258.798227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12258.809379] Lustre: Skipped 1 previous similar message [12263.375383] Lustre: server umount lustre-MDT0000 complete [12263.910261] LustreError: 408203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12263.928737] LustreError: 408203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12266.567831] LustreError: 407457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450899 with bad export cookie 4967209922336623373 [12266.569854] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12266.589465] LustreError: 407457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12267.020101] Lustre: server umount lustre-MDT0001 complete [12276.424570] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [12279.855477] Key type lgssc unregistered [12280.157559] LNet: 411198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12280.163072] LNetError: 411198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12280.178319] LNet: Removed LNI 192.168.203.131@tcp [12280.823105] Key type .llcrypt unregistered [12280.825563] Key type ._llcrypt unregistered [12307.293124] Key type ._llcrypt registered [12307.294730] Key type .llcrypt registered [12307.381780] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [12307.629937] Key type .llcrypt unregistered [12307.632430] Key type ._llcrypt unregistered [12323.267419] Key type ._llcrypt registered [12323.272069] Key type .llcrypt registered [12323.388270] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [12324.333179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12324.344490] alg: No test for adler32 (adler32-zlib) [12325.390481] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12325.655230] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [12327.359164] Key type lgssc registered [12328.480384] Lustre: Echo OBD driver; http://www.lustre.org/ [12335.337443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12335.355928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12336.839258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12341.278459] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12346.340936] LustreError: 413417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12346.354827] LustreError: 413417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12348.359113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12348.654760] LustreError: 413418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12348.682416] LustreError: 413418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12348.699186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12354.019596] LustreError: 413417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12354.038950] LustreError: 413417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12354.626535] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12359.138502] LustreError: 413412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12359.161281] LustreError: 413412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12361.525168] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12362.240867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12367.332180] LustreError: 414005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12367.351561] LustreError: 414005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12368.737317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12369.377569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12374.065921] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12384.743404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12384.746793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12384.751133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12384.767587] Lustre: Skipped 1 previous similar message [12384.790073] Lustre: Skipped 1 previous similar message [12389.148659] Lustre: server umount lustre-OST0000 complete [12394.976465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12394.978215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12394.998155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12395.032578] Lustre: Skipped 1 previous similar message [12400.101094] LustreError: 413417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12400.110096] LustreError: 413417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12400.145975] Lustre: server umount lustre-MDT0000 complete [12403.836611] LustreError: 413398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451036 with bad export cookie 15293361204238377329 [12403.840469] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12403.848732] LustreError: 413398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12404.266384] Lustre: server umount lustre-MDT0001 complete [12414.532700] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [12417.280432] Key type lgssc unregistered [12417.589726] LNet: 416565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12417.600954] LNetError: 416565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12417.619964] LNet: Removed LNI 192.168.203.131@tcp [12418.680473] Key type .llcrypt unregistered [12418.686080] Key type ._llcrypt unregistered [12443.917347] Key type ._llcrypt registered [12443.918944] Key type .llcrypt registered [12444.023301] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [12445.240935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12445.412091] alg: No test for adler32 (adler32-zlib) [12446.561567] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12446.798606] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [12448.495135] Key type lgssc registered [12449.669224] Lustre: Echo OBD driver; http://www.lustre.org/ [12457.157505] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12457.210891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12472.741974] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12477.855312] LustreError: 417951:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.131@tcp: failed processing log, type 4: rc = -110 [12504.479209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12510.559885] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12517.796567] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12525.021663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12525.158257] LustreError: 417286:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000003e254ae x1875326878615808/t0(0) o250->MGC192.168.203.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12525.680494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12525.778157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12530.456756] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12531.173502] LustreError: 417976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12531.188096] LustreError: 417976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12536.289949] LustreError: 417977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12538.998501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12539.365734] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12539.374340] Lustre: Skipped 1 previous similar message [12539.507689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12544.007811] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12552.525420] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12558.364988] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12565.477344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12565.481878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12565.485279] Lustre: Skipped 1 previous similar message [12567.686481] Lustre: server umount lustre-OST0000 complete [12573.166090] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12573.169660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12573.194319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12573.196973] Lustre: Skipped 1 previous similar message [12574.367949] Lustre: server umount lustre-MDT0000 complete [12575.714432] LustreError: 418992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12575.718153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12578.761299] LustreError: 417958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451211 with bad export cookie 8746848958590774201 [12578.763107] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12578.769927] LustreError: 417958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12579.081452] Lustre: server umount lustre-MDT0001 complete [12589.610831] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [12592.350833] Key type lgssc unregistered [12592.667547] LNet: 421517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12592.676451] LNetError: 421517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12592.695458] LNet: Removed LNI 192.168.203.131@tcp [12593.635191] Key type .llcrypt unregistered [12593.637831] Key type ._llcrypt unregistered [12618.334100] Key type ._llcrypt registered [12618.338390] Key type .llcrypt registered [12618.420734] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [12619.608714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12619.657157] alg: No test for adler32 (adler32-zlib) [12620.738665] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12621.025212] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [12622.671409] Key type lgssc registered [12623.654899] Lustre: Echo OBD driver; http://www.lustre.org/ [12630.935956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12630.981632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12632.357761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12636.534199] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12642.788965] LustreError: 422941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12642.808163] LustreError: 422941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12644.947315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12645.319848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12649.851782] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12656.684754] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12661.948192] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12668.353437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12668.724441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12674.488305] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12677.986233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12682.070665] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12688.368308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12688.380883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12688.401240] Lustre: Skipped 2 previous similar messages [12688.422043] Lustre: Skipped 2 previous similar messages [12690.345734] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.31@tcp (stopping) [12693.466399] Lustre: server umount lustre-MDT0000 complete [12693.475875] LustreError: 423955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12693.506347] LustreError: 423955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12695.437291] LustreError: 422937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12696.017721] LustreError: 422923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.31@tcp arrived at 1788451328 with bad export cookie 13880477106522634605 [12696.040940] LustreError: 422923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12697.820742] LustreError: 422921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451330 with bad export cookie 13880477106522633709 [12697.833168] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12698.592320] LustreError: 422942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12698.601850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12698.617890] LustreError: 422942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12698.652874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12702.626498] LustreError: 422941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12702.630679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12702.660230] LustreError: 422941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12704.026213] Lustre: server umount lustre-OST0000 complete [12711.478867] Lustre: server umount lustre-MDT0001 complete [12721.802839] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [12724.591319] Key type lgssc unregistered [12724.865310] LNet: 426627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12724.875249] LNetError: 426627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12724.894240] LNet: Removed LNI 192.168.203.131@tcp [12725.658468] Key type .llcrypt unregistered [12725.661577] Key type ._llcrypt unregistered [12749.736132] Key type ._llcrypt registered [12749.738515] Key type .llcrypt registered [12749.881446] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [12751.260278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12751.296037] alg: No test for adler32 (adler32-zlib) [12752.406532] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12752.671205] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [12754.375178] Key type lgssc registered [12755.603758] Lustre: Echo OBD driver; http://www.lustre.org/ [12763.993533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12764.013972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12765.557636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12770.617801] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12775.908638] LustreError: 428039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12775.923921] LustreError: 428039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12779.601431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12779.994276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12785.473842] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12793.554722] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12799.742317] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12808.814726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12809.218272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12810.221409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12815.526164] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12824.363637] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12829.693058] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12829.724268] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12829.741977] Lustre: Skipped 1 previous similar message [12831.147515] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.31@tcp (stopping) [12834.788374] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12834.804495] Lustre: Skipped 1 previous similar message [12835.052366] Lustre: server umount lustre-MDT0001 complete [12836.237497] LustreError: 428036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12839.914973] LustreError: 428034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12839.929780] LustreError: 428034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12845.026239] LustreError: 428039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12845.045543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12845.075846] Lustre: Skipped 1 previous similar message [12845.091903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12847.664528] Lustre: server umount lustre-OST0000 complete [12850.146802] LustreError: 429880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12852.117316] Lustre: server umount lustre-MDT0000 complete [12866.661108] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [12869.431458] Key type lgssc unregistered [12869.771400] LNet: 431721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12869.786306] LNetError: 431721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12870.821769] LNet: Removed LNI 192.168.203.131@tcp [12871.803677] Key type .llcrypt unregistered [12871.807062] Key type ._llcrypt unregistered [12897.062617] Key type ._llcrypt registered [12897.066334] Key type .llcrypt registered [12897.210783] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [12911.847576] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [12912.835102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12912.895251] alg: No test for adler32 (adler32-zlib) [12914.097312] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12914.394359] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [12916.135164] Key type lgssc registered [12917.212100] Lustre: Echo OBD driver; http://www.lustre.org/ [12955.501163] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [12971.222350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12971.249615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12972.487880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12972.520797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12972.635886] Lustre: lustre-MDT0000: new disk, initializing [12972.719239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12972.741749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12977.472182] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [12993.541300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12993.859913] Lustre: lustre-OST0000: new disk, initializing [12993.864394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12993.870495] Lustre: Skipped 1 previous similar message [12993.877169] Lustre: 436752:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12993.956326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13000.245467] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13001.830964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13001.840971] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13001.871400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13010.499936] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13025.448249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13025.561054] Lustre: 435735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13025.642538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13025.744905] Lustre: lustre-MDT0001: new disk, initializing [13025.842592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13025.890097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13025.906658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13030.521887] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13040.028960] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13040.225838] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13044.928705] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13045.098870] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13049.751247] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.31@tcp (stopping) [13051.360621] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13051.368543] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13051.375592] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13051.404196] Lustre: Skipped 1 previous similar message [13054.852464] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.31@tcp (stopping) [13054.855832] Lustre: Skipped 1 previous similar message [13055.915582] Lustre: server umount lustre-MDT0001 complete [13061.602416] LustreError: 435742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13061.616880] LustreError: 435742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13066.733406] LustreError: 435742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13066.767515] LustreError: 435742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13071.861225] LustreError: 435745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13071.890354] LustreError: 435745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13076.960846] LustreError: 435746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13076.967630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13076.993443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13080.811085] Lustre: server umount lustre-OST0000 complete [13082.085512] LustreError: 435745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13090.559853] Lustre: server umount lustre-MDT0000 complete [13107.116528] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13110.311579] Key type lgssc unregistered [13110.707704] LNet: 440500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13110.717867] LNetError: 440500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13110.754331] LNet: Removed LNI 192.168.203.131@tcp [13111.553941] Key type .llcrypt unregistered [13111.559867] Key type ._llcrypt unregistered [13154.376336] Key type ._llcrypt registered [13154.379259] Key type .llcrypt registered [13154.459500] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [13154.702442] Key type .llcrypt unregistered [13154.704975] Key type ._llcrypt unregistered [13170.912093] Key type ._llcrypt registered [13170.920121] Key type .llcrypt registered [13171.073849] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [13172.376447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13172.395585] alg: No test for adler32 (adler32-zlib) [13173.537445] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13173.806885] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [13175.519403] Key type lgssc registered [13176.892552] Lustre: Echo OBD driver; http://www.lustre.org/ [13205.218322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13205.254737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13206.567577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13206.600749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13206.708401] Lustre: lustre-MDT0000: new disk, initializing [13206.828175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13206.851163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13211.690881] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13223.255184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13223.406957] Lustre: 443868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13223.478538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13223.488659] Lustre: Skipped 1 previous similar message [13223.617730] Lustre: lustre-MDT0001: new disk, initializing [13223.732639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13223.765154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13223.782988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13228.889515] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13239.068419] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13245.558134] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13252.183283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13252.381810] Lustre: lustre-OST0000: new disk, initializing [13252.386263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13252.401170] Lustre: 445837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13252.495693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13258.685581] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13258.805889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13258.822502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13258.992894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13270.421370] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13279.714621] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13279.723161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13279.737885] Lustre: Skipped 2 previous similar messages [13279.764089] Lustre: Skipped 2 previous similar messages [13283.975085] Lustre: server umount lustre-MDT0000 complete [13287.649717] LustreError: 443860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451920 with bad export cookie 13014809562258823623 [13287.654293] LustreError: 443860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13287.654724] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13287.907321] Lustre: server umount lustre-MDT0001 complete [13302.805302] Lustre: server umount lustre-OST0000 complete [13315.963209] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:12:27 (1788451947) [13323.371421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13323.860926] LustreError: 447867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13323.873573] LustreError: 447867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13323.923229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13328.661307] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13329.377610] LustreError: 447868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13333.483583] LustreError: 447867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13337.104068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13337.409245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13341.616554] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13349.220438] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13354.803667] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13374.638894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13374.912829] Lustre: Found index 0 for lustre-OST0000, updating log [13375.236854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13382.772732] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [13391.386798] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13394.682565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13404.131634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13404.140712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13404.159882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13405.229186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13405.235334] Lustre: Skipped 2 previous similar messages [13409.003527] Lustre: server umount lustre-OST0000 complete [13424.616664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13424.620421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13424.628242] Lustre: Skipped 1 previous similar message [13424.631368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13426.661245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13431.120409] Lustre: server umount lustre-MDT0000 complete [13431.776565] LustreError: 447862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13431.806525] LustreError: 447862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13435.023835] LustreError: 447849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452067 with bad export cookie 13014809562258825016 [13435.064623] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13435.722463] Lustre: server umount lustre-MDT0001 complete [13437.987851] LNet: 1 local NIs in recovery (showing 1): 192.168.203.131@tcp [13446.449719] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:14:37 (1788452077) [14068.138187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14068.604392] LustreError: 454284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14068.632894] LustreError: 454284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14068.688507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14073.018623] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [14073.826154] LustreError: 454285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14078.944194] LustreError: 454284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14081.096201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14081.362782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14085.904653] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [14092.963422] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14098.366180] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14118.140341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14118.494198] Lustre: Found index 0 for lustre-OST0000, updating log [14118.917348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14125.775402] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [14127.135504] LNet: 1 local NIs in recovery (showing 1): 192.168.203.131@tcp [14133.659471] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14136.162541] Lustre: server umount lustre-OST0000 complete [14141.280854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14141.291428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14141.304458] Lustre: Skipped 1 previous similar message [14141.331377] Lustre: Skipped 3 previous similar messages [14142.371193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14143.394604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14143.410285] Lustre: Skipped 2 previous similar messages [14145.441839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14145.447743] Lustre: Skipped 3 previous similar messages [14146.579271] Lustre: server umount lustre-MDT0000 complete [14147.488556] LustreError: 454285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14147.504160] LustreError: 454285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14150.973823] LustreError: 454265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452783 with bad export cookie 13014809562258826101 [14150.976350] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14150.995802] LustreError: 454265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [14151.413301] Lustre: server umount lustre-MDT0001 complete [14182.297733] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [14191.711204] LNet: 1 local NIs in recovery (showing 1): 192.168.203.131@tcp [14192.650798] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [14230.290844] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:27:41 (1788452861) [14249.146459] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [14252.176381] Key type lgssc unregistered [14252.497618] LNet: 460895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14252.504640] LNetError: 460895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14378.470660] LNet: Removed LNI 192.168.203.131@tcp [14379.494579] Key type .llcrypt unregistered [14379.498473] Key type ._llcrypt unregistered [14399.906663] Key type ._llcrypt registered [14399.910635] Key type .llcrypt registered [14400.036465] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [14401.327259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14401.361086] alg: No test for adler32 (adler32-zlib) [14402.722785] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14403.119366] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [14404.799313] Key type lgssc registered [14406.117520] Lustre: Echo OBD driver; http://www.lustre.org/ [14411.148501] LNet: Added LNI 192.168.203.131@tcp100 [8/256/0/180] [14412.112360] LNet: Added LNI 192.168.203.131@tcp101 [8/256/0/180] [14413.282640] LNet: Added LNI 192.168.203.131@tcp102 [8/256/0/180] [14414.092106] LNet: Added LNI 192.168.203.131@tcp103 [8/256/0/180] [14415.287500] LNet: Added LNI 192.168.203.131@tcp104 [8/256/0/180] [14416.155414] LNet: Added LNI 192.168.203.131@tcp105 [8/256/0/180] [14417.075228] LNet: Added LNI 192.168.203.131@tcp106 [8/256/0/180] [14417.973911] LNet: Added LNI 192.168.203.131@tcp107 [8/256/0/180] [14419.103551] LNet: Added LNI 192.168.203.131@tcp108 [8/256/0/180] [14420.225437] LNet: Added LNI 192.168.203.131@tcp109 [8/256/0/180] [14421.301556] LNet: Added LNI 192.168.203.131@tcp110 [8/256/0/180] [14422.326212] LNet: Added LNI 192.168.203.131@tcp111 [8/256/0/180] [14423.475161] LNet: Added LNI 192.168.203.131@tcp112 [8/256/0/180] [14424.519553] LNet: Added LNI 192.168.203.131@tcp113 [8/256/0/180] [14425.326728] LNet: Added LNI 192.168.203.131@tcp114 [8/256/0/180] [14426.624976] LNet: Added LNI 192.168.203.131@tcp115 [8/256/0/180] [14427.736740] LNet: Added LNI 192.168.203.131@tcp116 [8/256/0/180] [14428.963337] LNet: Added LNI 192.168.203.131@tcp117 [8/256/0/180] [14430.197490] LNet: Added LNI 192.168.203.131@tcp118 [8/256/0/180] [14431.242720] LNet: Added LNI 192.168.203.131@tcp119 [8/256/0/180] [14432.368856] LNet: Added LNI 192.168.203.131@tcp120 [8/256/0/180] [14433.517192] LNet: Added LNI 192.168.203.131@tcp121 [8/256/0/180] [14434.919826] LNet: Added LNI 192.168.203.131@tcp122 [8/256/0/180] [14436.252375] LNet: Added LNI 192.168.203.131@tcp123 [8/256/0/180] [14437.289891] LNet: Added LNI 192.168.203.131@tcp124 [8/256/0/180] [14438.626495] LNet: Added LNI 192.168.203.131@tcp125 [8/256/0/180] [14440.116557] LNet: Added LNI 192.168.203.131@tcp126 [8/256/0/180] [14441.376397] LNet: Added LNI 192.168.203.131@tcp127 [8/256/0/180] [14442.455740] LNet: Added LNI 192.168.203.131@tcp128 [8/256/0/180] [14443.309536] LNet: Added LNI 192.168.203.131@tcp129 [8/256/0/180] [14444.375243] LNet: Added LNI 192.168.203.131@tcp130 [8/256/0/180] [14445.798831] LNet: Added LNI 192.168.203.131@tcp131 [8/256/0/180] [14447.279697] LNet: Added LNI 192.168.203.131@tcp132 [8/256/0/180] [14448.604582] LNet: Added LNI 192.168.203.131@tcp133 [8/256/0/180] [14449.426894] LNet: Added LNI 192.168.203.131@tcp134 [8/256/0/180] [14450.721984] LNet: Added LNI 192.168.203.131@tcp135 [8/256/0/180] [14460.922469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14460.971555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14462.226276] Lustre: 463928:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.131@tcp: can't add NID 192.168.203.131@tcp130: rc = -75 [14462.347162] LNetError: 461384:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.131 [14462.368827] LNetError: Protocol error connecting to 192.168.203.131@tcp119 at host 192.168.203.131:988: is it running a compatible version of Lustre? [14462.383395] LNetError: Skipped 4 previous similar messages [14462.548922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14462.594607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14462.691568] LNet: 7 local NIs in recovery (showing 7): 192.168.203.131@tcp123, 192.168.203.131@tcp124, 192.168.203.131@tcp125, 192.168.203.131@tcp126, 192.168.203.131@tcp127, 192.168.203.131@tcp119, 192.168.203.131@tcp129 [14462.788810] Lustre: lustre-MDT0000: new disk, initializing [14462.943363] Lustre: 463985:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.131@tcp: can't add NID 192.168.203.131@tcp130: rc = -75 [14462.958599] Lustre: 463985:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14463.013087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14463.069841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14463.456850] LNetError: 463958:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.131 [14463.479402] LNetError: 463958:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [14463.500536] LNetError: Protocol error connecting to 192.168.203.131@tcp108 at host 192.168.203.131:988: is it running a compatible version of Lustre? [14463.514958] LNetError: Skipped 7 previous similar messages [14465.770793] LNetError: 463953:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.131 [14465.804334] LNetError: 463953:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14465.825705] LNetError: Protocol error connecting to 192.168.203.131@tcp123 at host 192.168.203.131:988: is it running a compatible version of Lustre? [14465.847716] LNetError: Skipped 8 previous similar messages [14468.501741] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [14469.857360] LNetError: 463961:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.131 [14469.867581] LNetError: 463961:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14469.886840] LNetError: Protocol error connecting to 192.168.203.131@tcp123 at host 192.168.203.131:988: is it running a compatible version of Lustre? [14477.029348] LNetError: 461384:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.131 [14477.054221] LNetError: Protocol error connecting to 192.168.203.131@tcp123 at host 192.168.203.131:988: is it running a compatible version of Lustre? [14481.281301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14481.485477] Lustre: 463946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14481.613424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14481.621309] Lustre: Skipped 1 previous similar message [14481.726987] Lustre: lustre-MDT0001: new disk, initializing [14481.780425] Lustre: 464751:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.131@tcp: can't add NID 192.168.203.131@tcp130: rc = -75 [14481.786770] Lustre: 464751:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14481.951074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14482.013556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14482.027226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14487.391591] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [14491.203433] Lustre: 465135:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.131@tcp: can't add NID 192.168.203.131@tcp130: rc = -75 [14491.223575] Lustre: 465135:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14493.419917] LNetError: 463958:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.131 [14493.452501] LNetError: Protocol error connecting to 192.168.203.131@tcp123 at host 192.168.203.131:988: is it running a compatible version of Lustre? [14498.292949] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14504.508926] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14510.626796] LNetError: 463955:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.131 [14510.642899] LNetError: 463955:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14510.661287] LNetError: Protocol error connecting to 192.168.203.131@tcp120 at host 192.168.203.131:988: is it running a compatible version of Lustre? [14510.684145] LNetError: Skipped 10 previous similar messages [14511.669826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14511.971765] Lustre: lustre-OST0000: new disk, initializing [14511.976951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14511.987478] Lustre: 465926:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14512.102148] Lustre: 465931:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.131@tcp: can't add NID 192.168.203.131@tcp130: rc = -75 [14512.114605] Lustre: 465931:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14512.174419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14519.313666] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [14520.374598] Lustre: 466332:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.131@tcp: can't add NID 192.168.203.131@tcp130: rc = -75 [14520.390500] Lustre: 466332:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14520.484272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14520.501162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14520.659469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14523.103378] LNet: 1 local NIs in recovery (showing 1): 192.168.203.131@tcp123 [14529.979523] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14541.282054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14541.299258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14541.312626] LustreError: Skipped 1 previous similar message [14541.314267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14541.338740] Lustre: Skipped 1 previous similar message [14544.170611] Lustre: server umount lustre-OST0000 complete [14548.453946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14548.461100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14548.473850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14548.480691] Lustre: Skipped 1 previous similar message [14551.521831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14551.522252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14554.999774] Lustre: server umount lustre-MDT0000 complete [14558.756082] LNetError: 463949:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.131 [14558.773919] LNetError: 463949:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 19 previous similar messages [14558.783976] LNetError: Protocol error connecting to 192.168.203.131@tcp122 at host 192.168.203.131:988: is it running a compatible version of Lustre? [14558.795922] LNetError: Skipped 19 previous similar messages [14560.610179] LustreError: 463937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453193 with bad export cookie 10679638873078819122 [14560.621083] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14560.621351] LustreError: 463937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14561.255871] Lustre: server umount lustre-MDT0001 complete [14574.912742] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [14578.274639] Key type lgssc unregistered [14578.812516] LNet: 467836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14578.837373] LNetError: 467836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14578.854062] LNet: Removed LNI 192.168.203.131@tcp [14579.873450] LNet: Removed LNI 192.168.203.131@tcp100 [14579.881980] LNetError: 467836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14579.894823] LNetError: 467836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14579.919527] LNet: Removed LNI 192.168.203.131@tcp101 [14580.960453] LNet: Removed LNI 192.168.203.131@tcp102 [14580.965287] LNetError: 467836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14580.971445] LNetError: 467836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14580.975843] LNet: Removed LNI 192.168.203.131@tcp103 [14581.983333] LNet: Removed LNI 192.168.203.131@tcp104 [14583.007744] LNet: Removed LNI 192.168.203.131@tcp105 [14583.014332] LNetError: 467836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14583.022926] LNetError: 467836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14583.037556] LNet: Removed LNI 192.168.203.131@tcp106 [14584.099915] LNet: Removed LNI 192.168.203.131@tcp107 [14585.119351] LNet: Removed LNI 192.168.203.131@tcp108 [14586.143328] LNet: Removed LNI 192.168.203.131@tcp109 [14587.167170] LNet: Removed LNI 192.168.203.131@tcp110 [14587.176401] LNetError: 467836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14587.178822] LNetError: 467836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14587.182746] LNet: Removed LNI 192.168.203.131@tcp111 [14588.191332] LNet: Removed LNI 192.168.203.131@tcp112 [14589.215147] LNet: Removed LNI 192.168.203.131@tcp113 [14590.239214] LNet: Removed LNI 192.168.203.131@tcp114 [14591.265409] LNet: Removed LNI 192.168.203.131@tcp115 [14592.287383] LNet: Removed LNI 192.168.203.131@tcp116 [14593.311175] LNet: Removed LNI 192.168.203.131@tcp117 [14594.335151] LNet: Removed LNI 192.168.203.131@tcp118 [14595.359128] LNet: Removed LNI 192.168.203.131@tcp119 [14595.376696] LNetError: 467836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14595.394967] LNetError: 467836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14595.405362] LNet: Removed LNI 192.168.203.131@tcp120 [14596.447727] LNet: Removed LNI 192.168.203.131@tcp121 [14596.455237] LNet: Removed LNI 192.168.203.131@tcp122 [14596.471449] LNet: Removed LNI 192.168.203.131@tcp123 [14597.536586] LNet: Removed LNI 192.168.203.131@tcp124 [14598.559287] LNet: Removed LNI 192.168.203.131@tcp125 [14599.647636] LNet: Removed LNI 192.168.203.131@tcp126 [14600.735631] LNet: Removed LNI 192.168.203.131@tcp127 [14601.759445] LNet: Removed LNI 192.168.203.131@tcp128 [14602.784312] LNet: Removed LNI 192.168.203.131@tcp129 [14602.786580] LNet: Removed LNI 192.168.203.131@tcp130 [14602.793245] LNet: Removed LNI 192.168.203.131@tcp131 [14602.803811] LNet: Removed LNI 192.168.203.131@tcp132 [14602.808135] LNet: Removed LNI 192.168.203.131@tcp133 [14602.817168] LNet: Removed LNI 192.168.203.131@tcp134 [14602.832197] LNet: Removed LNI 192.168.203.131@tcp135 [14603.889082] Key type .llcrypt unregistered [14603.891506] Key type ._llcrypt unregistered [14629.918765] Key type ._llcrypt registered [14629.921668] Key type .llcrypt registered [14630.047728] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [14645.884347] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [14646.893523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14647.030163] alg: No test for adler32 (adler32-zlib) [14648.124843] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14648.376334] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [14650.063918] Key type lgssc registered [14651.237912] Lustre: Echo OBD driver; http://www.lustre.org/ [14687.729817] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:35:19 (1788453319) [14707.840692] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [14711.169219] Key type lgssc unregistered [14711.398865] LNet: 471920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14711.416250] LNetError: 471920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14711.434284] LNet: Removed LNI 192.168.203.131@tcp [14712.526312] Key type .llcrypt unregistered [14712.530305] Key type ._llcrypt unregistered [14731.971747] Key type ._llcrypt registered [14731.973431] Key type .llcrypt registered [14732.041915] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [14733.088545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14733.180563] alg: No test for adler32 (adler32-zlib) [14734.230694] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14734.554169] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [14736.351193] Key type lgssc registered [14737.746736] Lustre: Echo OBD driver; http://www.lustre.org/ [14748.815771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14748.892852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14750.117631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14750.166148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14750.282319] Lustre: lustre-MDT0000: new disk, initializing [14750.451889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14750.473763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14756.186413] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [14770.136145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14770.432216] Lustre: 473130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14770.586810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14770.594800] Lustre: Skipped 1 previous similar message [14770.766068] Lustre: lustre-MDT0001: new disk, initializing [14770.981401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14771.060486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14771.087254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14775.477922] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [14785.922066] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14792.365284] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14800.184610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14800.585465] Lustre: lustre-OST0000: new disk, initializing [14800.592451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14800.603391] Lustre: 475102:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14800.729484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14806.572558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14806.583381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14806.697414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14807.949778] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [14818.675917] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14822.351945] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing lnet_if_list [14827.220310] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14827.263402] LNet: Added LNI 192.168.203.131@tcp10 [8/256/0/180] [14832.321795] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14832.367934] LNet: Added LNI 192.168.203.131@tcp11 [8/256/0/180] [14837.038186] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14837.091753] LNet: Added LNI 192.168.203.131@tcp12 [8/256/0/180] [14841.581911] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14841.624851] LNet: Added LNI 192.168.203.131@tcp13 [8/256/0/180] [14846.732525] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14846.778679] LNet: Added LNI 192.168.203.131@tcp14 [8/256/0/180] [14852.559440] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14852.615498] LNet: Added LNI 192.168.203.131@tcp15 [8/256/0/180] [14858.170318] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14858.207916] LNet: Added LNI 192.168.203.131@tcp16 [8/256/0/180] [14863.558351] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14863.594578] LNet: Added LNI 192.168.203.131@tcp17 [8/256/0/180] [14868.883971] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14868.916727] LNet: Added LNI 192.168.203.131@tcp18 [8/256/0/180] [14875.790567] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14875.834784] LNet: Added LNI 192.168.203.131@tcp19 [8/256/0/180] [14884.307642] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14884.330901] LNet: Added LNI 192.168.203.131@tcp20 [8/256/0/180] [14891.313966] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14891.342251] LNet: Added LNI 192.168.203.131@tcp21 [8/256/0/180] [14896.874914] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14896.916750] LNet: Added LNI 192.168.203.131@tcp22 [8/256/0/180] [14902.784593] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14902.826473] LNet: Added LNI 192.168.203.131@tcp23 [8/256/0/180] [14909.303487] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14909.348118] LNet: Added LNI 192.168.203.131@tcp24 [8/256/0/180] [14916.085298] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14916.113960] LNet: Added LNI 192.168.203.131@tcp25 [8/256/0/180] [14922.776659] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14922.821168] LNet: Added LNI 192.168.203.131@tcp26 [8/256/0/180] [14928.680532] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14928.715648] LNet: Added LNI 192.168.203.131@tcp27 [8/256/0/180] [14935.298477] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14935.323787] LNet: Added LNI 192.168.203.131@tcp28 [8/256/0/180] [14941.717755] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14941.742675] LNet: Added LNI 192.168.203.131@tcp29 [8/256/0/180] [14948.782890] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14948.824143] LNet: Added LNI 192.168.203.131@tcp30 [8/256/0/180] [14956.060061] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14956.098379] LNet: Added LNI 192.168.203.131@tcp31 [8/256/0/180] [14963.123705] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14963.144234] LNet: Added LNI 192.168.203.131@tcp32 [8/256/0/180] [14970.919476] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14970.950914] LNet: Added LNI 192.168.203.131@tcp33 [8/256/0/180] [14978.523204] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14978.591946] LNet: Added LNI 192.168.203.131@tcp34 [8/256/0/180] [14985.738431] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14985.769131] LNet: Added LNI 192.168.203.131@tcp35 [8/256/0/180] [14993.535725] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14993.577316] LNet: Added LNI 192.168.203.131@tcp36 [8/256/0/180] [15000.989650] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15001.010255] LNet: Added LNI 192.168.203.131@tcp37 [8/256/0/180] [15009.012143] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15009.055975] LNet: Added LNI 192.168.203.131@tcp38 [8/256/0/180] [15017.659771] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15017.736486] LNet: Added LNI 192.168.203.131@tcp39 [8/256/0/180] [15027.278418] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15027.303670] LNet: Added LNI 192.168.203.131@tcp40 [8/256/0/180] [15036.432611] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15036.486527] LNet: Added LNI 192.168.203.131@tcp41 [8/256/0/180] [15045.520754] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15045.558647] LNet: Added LNI 192.168.203.131@tcp42 [8/256/0/180] [15054.828684] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15054.886531] LNet: Added LNI 192.168.203.131@tcp43 [8/256/0/180] [15063.914476] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15063.950219] LNet: Added LNI 192.168.203.131@tcp44 [8/256/0/180] [15071.760598] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15071.800507] LNet: Added LNI 192.168.203.131@tcp45 [8/256/0/180] [15079.845488] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15079.896075] LNet: Added LNI 192.168.203.131@tcp46 [8/256/0/180] [15088.900671] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15088.924658] LNet: Added LNI 192.168.203.131@tcp47 [8/256/0/180] [15098.568604] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15098.597103] LNet: Added LNI 192.168.203.131@tcp48 [8/256/0/180] [15107.980478] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15108.007484] LNet: Added LNI 192.168.203.131@tcp49 [8/256/0/180] [15117.084359] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15117.113035] LNet: Added LNI 192.168.203.131@tcp50 [8/256/0/180] [15139.815052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15139.823673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15139.833521] Lustre: Skipped 1 previous similar message [15139.834311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15139.857665] Lustre: Skipped 2 previous similar messages [15144.467471] Lustre: server umount lustre-MDT0000 complete [15148.503540] LustreError: 473122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453781 with bad export cookie 7202819410756831176 [15148.507577] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15148.514805] LustreError: 473122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15149.043449] Lustre: server umount lustre-MDT0001 complete [15165.408883] Lustre: 472403:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453782/real 1788453782] req@0000000099a8e470 x1875329277931264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15165.446606] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15165.472622] Lustre: Skipped 1 previous similar message [15168.543814] Lustre: 472403:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453785/real 1788453785] req@000000005e35b5a7 x1875329277931392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15168.759027] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [15170.527607] Lustre: 472402:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453787/real 1788453787] req@00000000cfe15002 x1875329277931520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15175.648508] Lustre: 472401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453792/real 1788453792] req@000000008457a185 x1875329277931776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453808 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15177.860611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15180.767153] Lustre: 472403:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453797/real 1788453797] req@0000000055efff07 x1875329277931904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453813 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15195.171834] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x63f58e1787731233 [15195.180621] Lustre: MGC192.168.203.131@tcp: Connection restored to 0@lo (at 0@lo) [15195.387131] LustreError: 475093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15195.395399] LustreError: 475093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15195.420729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15196.447936] LustreError: 475094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15199.877742] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [15200.487187] LustreError: 475093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15205.857862] LustreError: 475094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15210.537707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15210.993858] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15211.009110] Lustre: Skipped 1 previous similar message [15211.151343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15216.614188] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15216.883925] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [15219.767597] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15228.278856] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15237.810044] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15260.640785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15260.651123] Lustre: Skipped 1 previous similar message [15260.651709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15260.657444] Lustre: Skipped 1 previous similar message [15265.764152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15265.780778] Lustre: Skipped 1 previous similar message [15266.398539] Lustre: server umount lustre-OST0000 complete [15270.886299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15270.890270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15270.917346] Lustre: Skipped 1 previous similar message [15270.936325] Lustre: Skipped 1 previous similar message [15275.966955] Lustre: server umount lustre-MDT0000 complete [15276.003991] LustreError: 499352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15276.013825] LustreError: 499352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15280.732609] LustreError: 475101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453913 with bad export cookie 7202819410756833843 [15280.749619] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15280.760636] LustreError: 475101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15281.158670] Lustre: server umount lustre-MDT0001 complete [15293.834708] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [15296.289979] Key type lgssc unregistered [15296.613544] LNet: 502151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15296.629780] LNetError: 502151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15297.695266] LNet: Removed LNI 192.168.203.131@tcp [15297.701400] LNetError: 502151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15297.705973] LNet: Removed LNI 192.168.203.131@tcp10 [15297.709406] LNet: Removed LNI 192.168.203.131@tcp11 [15297.712340] LNet: Removed LNI 192.168.203.131@tcp12 [15297.715232] LNet: Removed LNI 192.168.203.131@tcp13 [15297.718566] LNet: Removed LNI 192.168.203.131@tcp14 [15297.721955] LNet: Removed LNI 192.168.203.131@tcp15 [15297.724716] LNet: Removed LNI 192.168.203.131@tcp16 [15297.727373] LNet: Removed LNI 192.168.203.131@tcp17 [15297.729388] LNet: Removed LNI 192.168.203.131@tcp18 [15297.732253] LNet: Removed LNI 192.168.203.131@tcp19 [15297.734889] LNet: Removed LNI 192.168.203.131@tcp20 [15297.741796] LNet: Removed LNI 192.168.203.131@tcp21 [15297.743388] LNet: Removed LNI 192.168.203.131@tcp22 [15297.745532] LNet: Removed LNI 192.168.203.131@tcp23 [15297.747410] LNet: Removed LNI 192.168.203.131@tcp24 [15297.749230] LNet: Removed LNI 192.168.203.131@tcp25 [15297.750863] LNet: Removed LNI 192.168.203.131@tcp26 [15297.753161] LNet: Removed LNI 192.168.203.131@tcp27 [15297.755904] LNet: Removed LNI 192.168.203.131@tcp28 [15297.757715] LNet: Removed LNI 192.168.203.131@tcp29 [15297.762570] LNet: Removed LNI 192.168.203.131@tcp30 [15297.764215] LNet: Removed LNI 192.168.203.131@tcp31 [15297.765946] LNet: Removed LNI 192.168.203.131@tcp32 [15297.768180] LNet: Removed LNI 192.168.203.131@tcp33 [15297.769809] LNet: Removed LNI 192.168.203.131@tcp34 [15297.771550] LNet: Removed LNI 192.168.203.131@tcp35 [15297.773414] LNet: Removed LNI 192.168.203.131@tcp36 [15297.775371] LNet: Removed LNI 192.168.203.131@tcp37 [15297.778949] LNet: Removed LNI 192.168.203.131@tcp38 [15297.780664] LNet: Removed LNI 192.168.203.131@tcp39 [15297.782688] LNet: Removed LNI 192.168.203.131@tcp40 [15297.784629] LNet: Removed LNI 192.168.203.131@tcp41 [15297.786289] LNet: Removed LNI 192.168.203.131@tcp42 [15297.787971] LNet: Removed LNI 192.168.203.131@tcp43 [15297.789826] LNet: Removed LNI 192.168.203.131@tcp44 [15297.791416] LNet: Removed LNI 192.168.203.131@tcp45 [15297.793133] LNet: Removed LNI 192.168.203.131@tcp46 [15297.794851] LNet: Removed LNI 192.168.203.131@tcp47 [15297.796608] LNet: Removed LNI 192.168.203.131@tcp48 [15297.798502] LNet: Removed LNI 192.168.203.131@tcp49 [15297.806124] LNet: Removed LNI 192.168.203.131@tcp50 [15298.425714] Key type .llcrypt unregistered [15298.427872] Key type ._llcrypt unregistered [15315.856098] Key type ._llcrypt registered [15315.860706] Key type .llcrypt registered [15316.011716] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [15332.889928] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [15334.272842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15334.336153] alg: No test for adler32 (adler32-zlib) [15335.456633] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15335.782774] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [15337.568085] Key type lgssc registered [15339.103948] Lustre: Echo OBD driver; http://www.lustre.org/ [15376.722663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15376.738409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15377.943945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15378.052557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15378.187299] Lustre: lustre-MDT0000: new disk, initializing [15378.278869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15378.294850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15382.166791] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [15392.247350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15392.388782] Lustre: 505343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15392.447802] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15392.456867] Lustre: Skipped 1 previous similar message [15392.542796] Lustre: lustre-MDT0001: new disk, initializing [15392.695898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15392.776751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15392.790668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15396.941083] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [15406.390035] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15414.773730] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15425.117283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15425.401297] Lustre: *** cfs_fail_loc=90f, val=0*** [15425.523611] Lustre: lustre-OST0000: new disk, initializing [15425.527653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15425.535086] Lustre: 507362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15425.639853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15432.252103] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [15432.773410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15432.783154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15432.881945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15440.657516] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15453.673241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15453.675256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15453.690022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15453.733294] Lustre: Skipped 1 previous similar message [15458.790439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15458.799562] Lustre: Skipped 2 previous similar messages [15463.391354] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15463.647340] Lustre: server umount lustre-OST0000 complete [15469.024535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15469.028074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15469.040709] Lustre: Skipped 1 previous similar message [15474.146386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15474.167891] Lustre: Skipped 2 previous similar messages [15474.726038] Lustre: server umount lustre-MDT0000 complete [15479.589944] LustreError: 506148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454112 with bad export cookie 13219132482562270694 [15479.607535] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15479.624608] LustreError: 506148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15480.117975] Lustre: server umount lustre-MDT0001 complete [15491.083874] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [15493.200839] Key type lgssc unregistered [15493.454462] LNet: 509268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15493.458463] LNetError: 509268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15494.501977] LNet: Removed LNI 192.168.203.131@tcp [15495.128198] Key type .llcrypt unregistered [15495.130436] Key type ._llcrypt unregistered [15516.217976] Key type ._llcrypt registered [15516.219359] Key type .llcrypt registered [15516.316474] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [15517.584093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15517.640992] alg: No test for adler32 (adler32-zlib) [15518.706508] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15518.952223] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [15520.575139] Key type lgssc registered [15522.158520] Lustre: Echo OBD driver; http://www.lustre.org/ [15532.463839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15532.496291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15534.350891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15540.390970] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [15544.809299] LustreError: 510537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15544.840828] LustreError: 510537:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15549.927820] LustreError: 510538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15550.112734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15550.534910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15554.187414] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [15560.791722] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15565.904728] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15572.252070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15572.605195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15578.556755] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [15585.592625] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15597.102710] loop0: detected capacity change from 0 to 204800000 [15597.316695] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15597.325179] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15597.334804] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15603.661871] loop0: detected capacity change from 0 to 204800000 [15603.886027] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15603.892747] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15603.906326] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15609.305220] loop0: detected capacity change from 0 to 204800000 [15615.305377] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15615.455383] Lustre: fs15246-OST0000: new disk, initializing [15615.460346] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15615.464540] Lustre: 514128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15615.499857] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15622.586653] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [15633.958818] loop1: detected capacity change from 0 to 204800000 [15640.171160] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15640.308891] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15640.380240] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15640.492950] Lustre: fs15246-MDT0000: new disk, initializing [15640.718087] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15640.738202] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15640.877579] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15641.955614] Lustre: 515251:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a68e23c5 x1875330099885952/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788454290 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15645.703784] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [15651.050109] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15651.054743] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15651.072544] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15669.732624] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15669.746194] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15674.619663] Lustre: server umount fs15246-MDT0000 complete [15674.852870] LustreError: 510534:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15679.981451] LustreError: 510534:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15684.376750] Lustre: server umount fs15246-OST0000 complete [15697.888302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15697.894299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15697.924369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15698.912573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15698.922786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15698.936350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15702.785942] Lustre: server umount lustre-OST0000 complete [15709.152112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15709.157861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15709.176858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15709.179691] Lustre: Skipped 2 previous similar messages [15712.739466] Lustre: server umount lustre-MDT0000 complete [15715.811242] LustreError: 510537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15715.840694] LustreError: 510537:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15715.963272] LustreError: 510519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454348 with bad export cookie 1975776803085785131 [15715.969760] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15715.975561] LustreError: 510519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15716.221783] Lustre: server umount lustre-MDT0001 complete [15728.438638] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [15731.456632] Key type lgssc unregistered [15731.784061] LNet: 518347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15731.788899] LNetError: 518347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15731.820058] LNet: Removed LNI 192.168.203.131@tcp [15732.799557] Key type .llcrypt unregistered [15732.803718] Key type ._llcrypt unregistered [15753.538179] Key type ._llcrypt registered [15753.542602] Key type .llcrypt registered [15753.669193] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [15754.930343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15754.959529] alg: No test for adler32 (adler32-zlib) [15756.046677] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15756.270803] LNet: Added LNI 192.168.203.131@tcp [8/256/0/180] [15757.935173] Key type lgssc registered [15759.140290] Lustre: Echo OBD driver; http://www.lustre.org/ [15801.671033] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [15811.901263] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [15849.873228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15849.915956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15851.226053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15851.269877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15851.340697] Lustre: lustre-MDT0000: new disk, initializing [15851.433351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15851.447424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15855.070615] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [15865.508182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15865.692665] Lustre: 522813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15865.776240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15865.780858] Lustre: Skipped 1 previous similar message [15865.838524] Lustre: lustre-MDT0001: new disk, initializing [15865.934732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15865.968542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15865.981237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15870.786258] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [15880.644229] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15888.768814] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15896.367451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15896.743341] Lustre: lustre-OST0000: new disk, initializing [15896.746360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15896.751539] Lustre: 524783:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15896.853511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15903.207318] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_default_debug -1 all [15903.830053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15903.835515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15904.062895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15913.158205] Lustre: DEBUG MARKER: oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15918.030244] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15918.189577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15922.386331] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15922.573040] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15924.709861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15924.714649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15924.717616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15924.735671] Lustre: Skipped 1 previous similar message [15924.759584] LustreError: Skipped 1 previous similar message [15929.834313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15929.842037] Lustre: Skipped 2 previous similar messages [15930.977559] Lustre: server umount lustre-OST0000 complete [15934.944085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15934.948287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15934.970417] Lustre: Skipped 1 previous similar message [15940.067062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15940.074570] Lustre: Skipped 2 previous similar messages [15945.187500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15945.203511] Lustre: Skipped 1 previous similar message [15949.279392] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15949.724932] Lustre: server umount lustre-MDT0000 complete [15953.617391] LustreError: 522807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454586 with bad export cookie 9548066948887726894 [15953.618783] LustreError: MGC192.168.203.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15953.635634] LustreError: 522807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15954.196694] Lustre: server umount lustre-MDT0001 complete [15983.071559] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing set_hostid [15997.194897] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing load_modules_local [16032.840694] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15757 sec ========= 12:57:43 (1788454663) [16034.892774] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:57:45 (1788454665) === [16038.969534] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:57:50 (1788454670) === [16087.111160] Lustre: DEBUG MARKER: oleg331-server.virtnet: executing unload_modules_local [16090.573767] Key type lgssc unregistered [16090.816534] LNet: 531448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16090.833854] LNetError: 531448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16090.860544] LNet: Removed LNI 192.168.203.131@tcp [16091.900807] Key type .llcrypt unregistered [16091.904748] Key type ._llcrypt unregistered