[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 498428344 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524576K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002342] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007392] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010126] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013080] *** VALIDATE selinux *** [ 0.022452] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027639] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031111] *** VALIDATE tmpfs *** [ 0.033438] *** VALIDATE proc *** [ 0.035103] *** VALIDATE cgroup *** [ 0.036009] *** VALIDATE cgroup2 *** [ 0.037259] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039142] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041028] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.045054] debug: unmapping init [mem 0xffffffff9d859000-0xffffffff9d860fff] [ 0.048250] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049773] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050021] ... version: 2 [ 0.051012] ... bit width: 48 [ 0.052014] ... generic registers: 4 [ 0.053015] ... value mask: 0000ffffffffffff [ 0.054015] ... max period: 00007fffffffffff [ 0.055014] ... fixed-purpose events: 3 [ 0.056013] ... event mask: 000000070000000f [ 0.057278] rcu: Hierarchical SRCU implementation. [ 0.059482] smp: Bringing up secondary CPUs ... [ 0.060675] x86: Booting SMP configuration: [ 0.061027] .... node #0, CPUs: #1 #2 #3 [ 0.065141] smp: Brought up 1 node, 4 CPUs [ 0.067026] smpboot: Max logical packages: 1 [ 0.068016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143327] node 0 deferred pages initialised in 73ms [ 0.147106] devtmpfs: initialized [ 0.148275] x86/mm: Memory block size: 128MB [ 0.152833] gcov: version magic: 0x41383552 [ 0.153611] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160346] pinctrl core: initialized pinctrl subsystem [ 0.162193] [ 0.162769] ************************************************************* [ 0.165016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168023] ** ** [ 0.169010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171013] ** ** [ 0.173014] ** This means that this kernel is built to expose internal ** [ 0.176012] ** IOMMU data structures, which may compromise security on ** [ 0.178010] ** your system. ** [ 0.181019] ** ** [ 0.183010] ** If you see this message and you are not debugging the ** [ 0.185013] ** kernel, report this immediately to your vendor! ** [ 0.187016] ** ** [ 0.190013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192010] ************************************************************* [ 0.195500] NET: Registered protocol family 16 [ 0.197508] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.205031] cpuidle: using governor menu [ 0.206775] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208419] PCI: Using configuration type 1 for base access [ 0.210131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219113] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224082] cryptd: max_cpu_qlen set to 1000 [ 0.227244] ACPI: Added _OSI(Module Device) [ 0.229012] ACPI: Added _OSI(Processor Device) [ 0.230019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236306] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242647] ACPI: Interpreter enabled [ 0.243048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245018] ACPI: Using IOAPIC for interrupt routing [ 0.246083] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.248430] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258609] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270343] acpiphp: Slot [2] registered [ 0.272085] acpiphp: Slot [5] registered [ 0.273115] acpiphp: Slot [6] registered [ 0.275105] acpiphp: Slot [7] registered [ 0.276110] acpiphp: Slot [8] registered [ 0.278136] acpiphp: Slot [9] registered [ 0.280090] acpiphp: Slot [10] registered [ 0.281104] acpiphp: Slot [3] registered [ 0.283104] acpiphp: Slot [4] registered [ 0.285144] acpiphp: Slot [11] registered [ 0.286149] acpiphp: Slot [12] registered [ 0.288159] acpiphp: Slot [13] registered [ 0.289113] acpiphp: Slot [14] registered [ 0.291114] acpiphp: Slot [15] registered [ 0.293087] acpiphp: Slot [16] registered [ 0.294147] acpiphp: Slot [17] registered [ 0.296201] acpiphp: Slot [18] registered [ 0.297093] acpiphp: Slot [19] registered [ 0.298077] acpiphp: Slot [20] registered [ 0.299073] acpiphp: Slot [21] registered [ 0.301069] acpiphp: Slot [22] registered [ 0.303110] acpiphp: Slot [23] registered [ 0.305111] acpiphp: Slot [24] registered [ 0.307126] acpiphp: Slot [25] registered [ 0.308135] acpiphp: Slot [26] registered [ 0.310106] acpiphp: Slot [27] registered [ 0.311097] acpiphp: Slot [28] registered [ 0.313099] acpiphp: Slot [29] registered [ 0.315106] acpiphp: Slot [30] registered [ 0.317120] acpiphp: Slot [31] registered [ 0.318082] PCI host bridge to bus 0000:00 [ 0.320023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.323021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.326021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.329023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.332031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.337208] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.339994] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.343458] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.354842] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.358937] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.362017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.367016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.370323] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373803] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.376044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.379782] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.386020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.400013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.405013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.411645] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.418016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.426013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.448013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.464013] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.473014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.480015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.498014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.508348] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.516013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.522013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.541025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.552095] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.561015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.573017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.595014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.607412] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.614020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.621024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.637021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.647523] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.654015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.660015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.676017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.688207] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.691495] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.694567] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.698552] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.701251] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.705385] iommu: Default domain type: Passthrough [ 0.709545] SCSI subsystem initialized [ 0.711238] ACPI: bus type USB registered [ 0.714168] usbcore: registered new interface driver usbfs [ 0.718105] usbcore: registered new interface driver hub [ 0.721099] usbcore: registered new device driver usb [ 0.724191] pps_core: LinuxPPS API ver. 1 registered [ 0.727010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.730158] PTP clock support registered [ 0.733150] EDAC MC: Ver: 3.0.0 [ 0.735115] PCI: Using ACPI for IRQ routing [ 0.736905] NetLabel: Initializing [ 0.738010] NetLabel: domain hash size = 128 [ 0.739009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.741081] NetLabel: unlabeled traffic allowed by default [ 0.743163] vgaarb: loaded [ 0.744306] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.746014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.751338] clocksource: Switched to clocksource kvm-clock [ 0.863850] VFS: Disk quotas dquot_6.6.0 [ 0.865182] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.867555] *** VALIDATE ramfs *** [ 0.868772] *** VALIDATE hugetlbfs *** [ 0.870737] pnp: PnP ACPI init [ 0.873031] pnp: PnP ACPI: found 6 devices [ 0.898363] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.901786] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.904100] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.906493] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.909038] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.911698] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.914769] NET: Registered protocol family 2 [ 0.917142] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.923501] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.929181] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.936234] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.940237] TCP: Hash tables configured (established 65536 bind 65536) [ 0.943374] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.946914] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.950421] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.955087] NET: Registered protocol family 1 [ 0.958430] RPC: Registered named UNIX socket transport module. [ 0.960400] RPC: Registered udp transport module. [ 0.962018] RPC: Registered tcp transport module. [ 0.963585] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.965516] NET: Registered protocol family 44 [ 0.966993] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.968982] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.970864] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.972947] PCI: CLS 0 bytes, default 64 [ 0.974460] Unpacking initramfs... [ 2.416705] debug: unmapping init [mem 0xffff9012bcc54000-0xffff9012bffbffff] [ 2.420441] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.422182] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.424698] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.930647] Initialise system trusted keyrings [ 2.932507] Key type blacklist registered [ 2.934522] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.943106] zbud: loaded [ 2.946033] *** VALIDATE nfs *** [ 2.947423] *** VALIDATE nfs4 *** [ 2.948986] pstore: using deflate compression [ 2.952944] Platform Keyring initialized [ 3.054717] NET: Registered protocol family 38 [ 3.056176] Key type asymmetric registered [ 3.057778] Asymmetric key parser 'x509' registered [ 3.059705] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.062275] io scheduler mq-deadline registered [ 3.063907] io scheduler kyber registered [ 3.065478] io scheduler bfq registered [ 3.066990] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.069816] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.072922] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.075889] ACPI: Power Button [PWRF] [ 3.082880] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.092565] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.111642] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.124495] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.148921] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.180221] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.208668] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.213527] Non-volatile memory driver v1.3 [ 3.214730] Linux agpgart interface v0.103 [ 3.243150] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.247467] vda: detected capacity change from 0 to 74702848 [ 3.265591] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.269296] vdb: detected capacity change from 0 to 1073741824 [ 3.285247] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.288110] vdc: detected capacity change from 0 to 2621440000 [ 3.302515] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.305050] vdd: detected capacity change from 0 to 2621440000 [ 3.319803] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.323039] vde: detected capacity change from 0 to 4294967296 [ 3.340772] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.344245] vdf: detected capacity change from 0 to 4294967296 [ 3.360526] libphy: Fixed MDIO Bus: probed [ 3.367275] usbcore: registered new interface driver usbserial_generic [ 3.369382] usbserial: USB Serial support registered for generic [ 3.371264] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.375456] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.376947] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.379320] mousedev: PS/2 mouse device common for all mice [ 3.382059] rtc_cmos 00:05: RTC can wake from S4 [ 3.385287] rtc_cmos 00:05: registered as rtc0 [ 3.387174] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.387779] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.394454] intel_pstate: CPU model not supported [ 3.397938] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.401788] hid: raw HID events driver (C) Jiri Kosina [ 3.405442] usbcore: registered new interface driver usbhid [ 3.408203] usbhid: USB HID core driver [ 3.410212] drop_monitor: Initializing network drop monitor service [ 3.410221] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.413062] Initializing XFRM netlink socket [ 3.418219] NET: Registered protocol family 10 [ 3.420837] Segment Routing with IPv6 [ 3.422156] NET: Registered protocol family 17 [ 3.424490] mpls_gso: MPLS GSO support [ 3.431492] RAS: Correctable Errors collector initialized. [ 3.433273] AVX version of gcm_enc/dec engaged. [ 3.434608] AES CTR mode by8 optimization enabled [ 3.510219] sched_clock: Marking stable (3510187512, 0)->(4423468482, -913280970) [ 3.513772] registered taskstats version 1 [ 3.516065] Loading compiled-in X.509 certificates [ 3.518194] zswap: loaded using pool lzo/zbud [ 3.540193] Key type big_key registered [ 3.551780] Key type encrypted registered [ 3.553737] ima: No TPM chip found, activating TPM-bypass! [ 3.555752] ima: Allocated hash algorithm: sha1 [ 3.557702] ima: No architecture policies found [ 3.559758] evm: Initialising EVM extended attributes: [ 3.561887] evm: security.selinux [ 3.563128] evm: security.ima [ 3.563973] evm: security.capability [ 3.565178] evm: HMAC attrs: 0x1 [ 3.567330] rtc_cmos 00:05: setting system clock to 2026-08-15 16:47:50 UTC (1786812470) [ 3.572742] debug: unmapping init [mem 0xffffffff9e803000-0xffffffff9e9fffff] [ 3.575775] debug: unmapping init [mem 0xffffffff9d582000-0xffffffff9d858fff] [ 3.584146] Write protecting the kernel read-only data: 28672k [ 3.588175] debug: unmapping init [mem 0xffffffff9bc03000-0xffffffff9bdfffff] [ 3.591347] debug: unmapping init [mem 0xffffffff9c514000-0xffffffff9c5fffff] [ 3.622406] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.631774] systemd[1]: Detected virtualization kvm. [ 3.633665] systemd[1]: Detected architecture x86-64. [ 3.635701] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.660719] systemd[1]: No hostname configured. [ 3.662486] systemd[1]: Set hostname to . [ 3.664325] random: systemd: uninitialized urandom read (16 bytes read) [ 3.666528] systemd[1]: Initializing machine ID from random generator. [ 3.782447] random: systemd: uninitialized urandom read (16 bytes read) [ 3.785882] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.793187] random: systemd: uninitialized urandom read (16 bytes read) [ 3.796600] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.801558] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.378347] device-mapper: uevent: version 1.0.3 [ 4.380500] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.013921] virtio_net virtio0 ens2: renamed from eth0 [ 5.015424] random: fast init done [ 5.109521] scsi host0: ata_piix [ 5.121207] scsi host1: ata_piix [ 5.122520] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.124822] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.300224] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.069503] random: crng init done [ 10.070740] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.376120] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.454398] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.678068] SELinux: Disabled at runtime. [ 11.747327] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.756609] systemd[1]: Detected virtualization kvm. [ 11.758182] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.210528] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.214071] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.219808] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.227293] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.229676] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.234802] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.241514] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ 12.319855] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.629961] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.961300] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.962606] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.072344] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.089636] EDAC sbridge: Ver: 1.1.2 [ 14.406319] Key type dns_resolver registered [ 14.698956] NFS: Registering the id_resolver key type [ 14.700747] Key type id_resolver registered [ 14.702012] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg327-server login: [ 40.854378] libcfs: loading out-of-tree module taints kernel. [ 40.872736] Key type ._llcrypt registered [ 40.874588] Key type .llcrypt registered [ 40.922514] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 50.568812] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 52.056813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.068153] alg: No test for adler32 (adler32-zlib) [ 53.602123] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 54.455274] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 56.263590] Key type lgssc registered [ 58.501925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 78.860050] hrtimer: interrupt took 3582892 ns [ 79.311350] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 125.828654] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 140.913749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 140.968372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.250904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 142.273815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 142.370843] Lustre: lustre-MDT0000: new disk, initializing [ 142.458020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.471677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 146.905964] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.334581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.498630] Lustre: 6498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 160.576836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 160.581913] Lustre: Skipped 1 previous similar message [ 160.723522] Lustre: lustre-MDT0001: new disk, initializing [ 160.806343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 160.855512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 160.867359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 165.012588] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.842771] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 180.253855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 180.513873] Lustre: lustre-OST0000: new disk, initializing [ 180.518565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 180.532678] Lustre: 8434:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 180.601491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.931265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 186.958657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 187.056071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 187.171590] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.292845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.465300] Lustre: lustre-OST0001: new disk, initializing [ 201.471484] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 201.487640] Lustre: 9506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.572953] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 207.686615] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.500122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 210.512384] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 210.547838] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.071349] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 227.897230] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 234.183232] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing check_logdir /tmp/testlogs/ [ 239.434811] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing yml_node [ 244.714749] Lustre: DEBUG MARKER: Client: 2.17.57.2 [ 247.745245] Lustre: DEBUG MARKER: MDS: 2.17.57.2 [ 250.312691] Lustre: DEBUG MARKER: OSS: 2.17.57.2 [ 251.990479] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 12:51:57 EDT 2026 [ 268.226865] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 269.754959] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 287.200953] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.207570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.224647] Lustre: Skipped 3 previous similar messages [ 287.231136] Lustre: Skipped 2 previous similar messages [ 290.827673] Lustre: server umount lustre-MDT0000 complete [ 294.068909] LustreError: 6490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812761 with bad export cookie 12822958444270230769 [ 294.079760] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 294.086411] LustreError: 6490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 297.440326] LustreError: 6503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.441366] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.444509] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.444518] Lustre: Skipped 1 previous similar message [ 297.461557] LustreError: 6503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 297.474445] Lustre: Skipped 1 previous similar message [ 299.489597] LustreError: 6505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.490855] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.500541] LustreError: 6505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 299.539552] Lustre: Skipped 2 previous similar messages [ 300.447606] Lustre: server umount lustre-MDT0001 complete [ 303.872357] Lustre: server umount lustre-OST0000 complete [ 307.966726] Lustre: server umount lustre-OST0001 complete [ 316.607746] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 327.864466] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 367.268538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.617856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 367.654092] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 367.807429] Lustre: lustre-MDT0000: new disk, initializing [ 367.908368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 367.930206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 373.980607] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 387.136960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.442988] Lustre: 15060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 387.530403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 387.534653] Lustre: Skipped 1 previous similar message [ 387.600387] Lustre: lustre-MDT0001: new disk, initializing [ 387.786091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 387.921648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 387.940329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 392.654462] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 403.525583] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 410.780965] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 418.262463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 418.716079] Lustre: lustre-OST0000: new disk, initializing [ 418.719310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 418.737734] Lustre: 17030:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 418.864118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.953806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 419.974616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 420.121400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 425.693948] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 436.012973] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 441.814606] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 442.010964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 446.950744] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 447.232332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 451.039781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 451.045120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.064080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 451.086377] Lustre: Skipped 1 previous similar message [ 455.526598] Lustre: server umount lustre-OST0000 complete [ 459.232855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 459.240899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.257298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 459.260853] Lustre: Skipped 1 previous similar message [ 465.630852] Lustre: server umount lustre-MDT0000 complete [ 466.401351] LustreError: 15072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.410626] LustreError: 15072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 469.946823] LustreError: 15054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812936 with bad export cookie 12822958444270232302 [ 469.968387] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.395730] Lustre: server umount lustre-MDT0001 complete [ 475.806095] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:55:40 (1786812940) [ 478.641588] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 480.548818] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:55:45 (1786812945) [ 487.113193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.609988] LustreError: 19441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.751511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.121887] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 493.037031] LustreError: 19442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.359224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.669661] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 512.026593] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 517.285678] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 524.290881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 530.662169] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 538.378506] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 548.322851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.337349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 548.347393] Lustre: Skipped 2 previous similar messages [ 548.359292] Lustre: Skipped 3 previous similar messages [ 552.833500] Lustre: server umount lustre-OST0000 complete [ 558.562692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.389942] Lustre: server umount lustre-MDT0000 complete [ 563.694510] LustreError: 19442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.732676] LustreError: 19442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 567.217983] LustreError: 19421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813034 with bad export cookie 12822958444270233184 [ 567.230753] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 567.234064] LustreError: 19421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 567.687115] Lustre: server umount lustre-MDT0001 complete [ 579.220523] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 582.207472] Key type lgssc unregistered [ 582.536504] LNet: 22976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.542900] LNetError: 22976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.561390] LNet: Removed LNI 192.168.203.127@tcp [ 584.099864] Key type .llcrypt unregistered [ 584.102262] Key type ._llcrypt unregistered [ 606.820349] Key type ._llcrypt registered [ 606.822815] Key type .llcrypt registered [ 606.899670] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 608.306763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 608.347150] alg: No test for adler32 (adler32-zlib) [ 609.424867] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 609.611321] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 611.279180] Key type lgssc registered [ 612.219698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 621.208095] loop: module loaded [ 621.295833] loop0: detected capacity change from 0 to 204800000 [ 621.558328] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 621.586717] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 622.013651] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 630.470376] loop0: detected capacity change from 0 to 204800000 [ 630.681705] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 630.691510] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 630.701038] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.093560] loop0: detected capacity change from 0 to 204800000 [ 642.579038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 642.633423] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.014266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 644.094354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 644.185932] Lustre: lustre-MDT0000: new disk, initializing [ 644.378569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 644.418140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 648.788935] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 657.903880] loop1: detected capacity change from 0 to 204800000 [ 663.365555] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 663.610878] Lustre: lustre-OST07c6: new disk, initializing [ 663.619402] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 663.626406] Lustre: Skipped 1 previous similar message [ 663.630849] Lustre: 26164:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 663.711983] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 668.992551] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 671.340496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 671.346756] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 671.400808] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 673.045800] Lustre: 25154:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 673.054229] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 673.062057] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 673.068053] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 673.086486] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 673.094469] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 693.255190] Lustre: 25154:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 693.265183] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 693.278039] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 693.299838] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 693.308257] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 693.319235] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 698.396810] Lustre: 25154:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 698.415463] Lustre: 25154:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 698.425522] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 698.460433] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.486428] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 698.498988] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.511528] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 698.535285] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.557350] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 698.577786] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.610604] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 698.629595] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.886776] Lustre: 25154:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 704.903431] Lustre: 25154:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 704.915392] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 704.931777] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.942173] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 704.953196] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.969822] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 704.989067] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.998970] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 705.007889] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 705.019145] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 705.031062] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 711.845970] Lustre: 25154:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 711.876337] Lustre: 25154:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 711.899432] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 711.916320] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 711.935330] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 711.953410] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 711.977901] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 711.982171] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 711.986249] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 711.989665] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 711.993258] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 712.000464] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 725.156608] Lustre: 25152:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 725.178136] Lustre: 25152:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 725.193622] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 725.204033] Lustre: 25152:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 725.220778] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.256302] Lustre: 25152:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 725.273236] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 725.286793] Lustre: 25152:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 725.296877] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 725.304774] Lustre: 25152:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 725.322138] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.347713] Lustre: 25152:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 771.448153] Lustre: server umount lustre-OST07c6 complete [ 791.171571] Lustre: server umount lustre-MDT0000 complete [ 818.154194] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 821.408309] Key type lgssc unregistered [ 821.593970] LNet: 28761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.636888] LNetError: 28761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.662165] LNet: Removed LNI 192.168.203.127@tcp [ 823.017263] Key type .llcrypt unregistered [ 823.019074] Key type ._llcrypt unregistered [ 841.546379] Key type ._llcrypt registered [ 841.549968] Key type .llcrypt registered [ 841.643970] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 842.418491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 842.450257] alg: No test for adler32 (adler32-zlib) [ 843.724434] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 843.993884] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 845.775193] Key type lgssc registered [ 847.243101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 854.423502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 854.447853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.992439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.863346] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 865.255232] LustreError: 30051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 865.273129] LustreError: 30051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 868.699416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.037832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 873.758749] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 882.382596] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 888.492051] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 896.837618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 897.400211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.476536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 904.427710] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 913.395081] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 931.297359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 931.307295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.336509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 934.376621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 934.378652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 934.389240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 936.068606] Lustre: server umount lustre-OST0000 complete [ 940.512614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 940.519566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 940.547223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 940.559469] Lustre: Skipped 1 previous similar message [ 944.610176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 944.610476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.735071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 949.750083] Lustre: Skipped 2 previous similar messages [ 954.335117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 954.789086] Lustre: server umount lustre-MDT0000 complete [ 954.854692] LustreError: 30050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 954.918861] LustreError: 30050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 958.934916] LustreError: 30032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813425 with bad export cookie 17171662457778142167 [ 958.941792] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 958.952246] LustreError: 30032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 959.472249] Lustre: server umount lustre-MDT0001 complete [ 972.207144] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 974.933643] Key type lgssc unregistered [ 975.231231] LNet: 33790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.237548] LNetError: 33790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.250953] LNet: Removed LNI 192.168.203.127@tcp [ 976.268649] Key type .llcrypt unregistered [ 976.271702] Key type ._llcrypt unregistered [ 995.614223] Key type ._llcrypt registered [ 995.616861] Key type .llcrypt registered [ 995.705582] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 996.890366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 996.907015] alg: No test for adler32 (adler32-zlib) [ 997.934516] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 998.147741] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 999.831407] Key type lgssc registered [ 1001.567247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1009.000971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1009.036746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1010.410387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1015.550222] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1020.897521] LustreError: 35060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.944767] LustreError: 35060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1025.204492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.715630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1031.046094] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1038.514731] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1044.425817] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1051.673775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1051.970290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1056.052081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1057.382672] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1064.409311] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1080.802451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1080.806060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1080.814906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1080.819654] Lustre: Skipped 1 previous similar message [ 1080.831357] Lustre: Skipped 1 previous similar message [ 1084.347105] Lustre: server umount lustre-OST0000 complete [ 1091.041981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1091.052824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1091.077073] Lustre: Skipped 1 previous similar message [ 1091.094329] Lustre: Skipped 1 previous similar message [ 1094.790292] Lustre: server umount lustre-MDT0000 complete [ 1096.163331] LustreError: 35054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1096.200074] LustreError: 35054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1100.267527] LustreError: 35055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1100.308073] LustreError: 35055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1100.611061] LustreError: 35041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813567 with bad export cookie 11101078803568987317 [ 1100.618488] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1100.627220] LustreError: 35041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1101.690719] Lustre: server umount lustre-MDT0001 complete [ 1118.017518] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1121.412093] Key type lgssc unregistered [ 1121.669600] LNet: 38620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.681148] LNetError: 38620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1121.698703] LNet: Removed LNI 192.168.203.127@tcp [ 1122.414134] Key type .llcrypt unregistered [ 1122.418036] Key type ._llcrypt unregistered [ 1146.818923] Key type ._llcrypt registered [ 1146.823726] Key type .llcrypt registered [ 1146.923530] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1147.922365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1148.014822] alg: No test for adler32 (adler32-zlib) [ 1148.988369] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 1149.233961] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1150.903415] Key type lgssc registered [ 1151.877883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1158.655866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1158.687846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.036891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.054770] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1169.387794] LustreError: 40037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1169.425250] LustreError: 40037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1173.591434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.125665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1179.132640] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1186.089260] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1191.439507] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1197.281945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1197.572070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1203.104282] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1204.726617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1210.476935] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1217.332596] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 1218.490128] Lustre: server umount lustre-MDT0000 complete [ 1220.064891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1220.069295] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.075553] LustreError: 40766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1220.086650] Lustre: Skipped 1 previous similar message [ 1220.115348] LustreError: 40766:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1221.905541] LustreError: 40019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813688 with bad export cookie 9762462507170743269 [ 1221.911490] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1221.918106] LustreError: 40019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1222.414852] LustreError: 40032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.454716] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.27@tcp (stopping) [ 1225.184971] LustreError: 40034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1225.187050] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.217755] Lustre: Skipped 1 previous similar message [ 1225.224398] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1227.542559] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.27@tcp (stopping) [ 1227.552612] Lustre: Skipped 1 previous similar message [ 1228.205923] Lustre: server umount lustre-MDT0001 complete [ 1248.980980] Lustre: server umount lustre-OST0000 complete [ 1266.538244] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1268.885442] Key type lgssc unregistered [ 1269.269121] LNet: 43899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.280795] LNetError: 43899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.288391] LNet: Removed LNI 192.168.203.127@tcp [ 1270.073485] Key type .llcrypt unregistered [ 1270.075668] Key type ._llcrypt unregistered [ 1295.517841] Key type ._llcrypt registered [ 1295.521133] Key type .llcrypt registered [ 1295.639816] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1297.740549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1297.789459] alg: No test for adler32 (adler32-zlib) [ 1298.885709] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 1299.136687] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1300.895217] Key type lgssc registered [ 1301.932914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1311.230158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1311.273697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1312.808693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.014679] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1322.982839] LustreError: 45315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1322.995389] LustreError: 45315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1325.435594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.806890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1329.849876] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1336.903377] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1342.163411] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1348.862413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1349.240559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.267825] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1357.448787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1364.373340] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1371.969386] Lustre: lustre-OST0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 1373.156284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1373.157922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1373.197524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1373.238890] Lustre: Skipped 1 previous similar message [ 1377.038589] Lustre: lustre-OST0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 1377.727777] Lustre: server umount lustre-OST0000 complete [ 1397.728552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1397.756804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.801230] Lustre: Skipped 1 previous similar message [ 1397.838793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1398.757416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.889973] LustreError: 46371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.920169] LustreError: 46371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1404.097301] Lustre: server umount lustre-MDT0000 complete [ 1408.996354] LustreError: 45315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.012957] LustreError: 45296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813875 with bad export cookie 13398794077928417562 [ 1409.021612] LustreError: 45315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1409.054266] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1415.402943] Lustre: server umount lustre-MDT0001 complete [ 1426.401996] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1429.066603] Key type lgssc unregistered [ 1429.340366] LNet: 49027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1429.349112] LNetError: 49027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1429.375881] LNet: Removed LNI 192.168.203.127@tcp [ 1430.317715] Key type .llcrypt unregistered [ 1430.319725] Key type ._llcrypt unregistered [ 1453.816694] Key type ._llcrypt registered [ 1453.818499] Key type .llcrypt registered [ 1453.899808] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1455.205965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1455.256106] alg: No test for adler32 (adler32-zlib) [ 1456.389604] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 1456.638484] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1458.319425] Key type lgssc registered [ 1459.537788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1469.965142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1469.991029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1471.490444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.557613] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1481.701355] LustreError: 50444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1481.740395] LustreError: 50444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1484.258514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.872582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1488.689964] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1496.398733] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1502.014387] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1508.133349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1508.429470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1513.802893] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1516.671985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1523.079828] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1529.449730] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1534.165318] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1546.783305] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1513h13m31s [ 1547.986191] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1551.137905] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 1551.330685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1551.348416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1552.203894] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 1552.217239] Lustre: Skipped 3 previous similar messages [ 1556.330631] Lustre: server umount lustre-MDT0000 complete [ 1557.266687] LustreError: 53082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.510698] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.527943] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1559.570758] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1563.416338] LustreError: 52686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1565.842462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.966573] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1566.209512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.279327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1567.275404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1570.305619] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1573.524294] Lustre: DEBUG MARKER: Wait for df (36729) ... [ 1574.997887] Lustre: DEBUG MARKER: done [ 1581.537307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1581.543366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.546601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1581.555990] Lustre: Skipped 3 previous similar messages [ 1581.565949] Lustre: Skipped 4 previous similar messages [ 1584.813627] Lustre: server umount lustre-OST0000 complete [ 1592.802689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1592.815628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1592.819648] LustreError: Skipped 1 previous similar message [ 1592.873571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1592.876468] Lustre: Skipped 3 previous similar messages [ 1594.469704] Lustre: server umount lustre-MDT0000 complete [ 1598.736698] LustreError: 50426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814065 with bad export cookie 14611053024526277169 [ 1598.750604] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1598.754574] LustreError: 50426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1599.255591] Lustre: server umount lustre-MDT0001 complete [ 1610.327814] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1613.394589] Key type lgssc unregistered [ 1613.770548] LNet: 55373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1613.782103] LNetError: 55373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1613.806776] LNet: Removed LNI 192.168.203.127@tcp [ 1614.827887] Key type .llcrypt unregistered [ 1614.835183] Key type ._llcrypt unregistered [ 1657.191210] Key type ._llcrypt registered [ 1657.196382] Key type .llcrypt registered [ 1657.321689] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1658.549457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1658.607480] alg: No test for adler32 (adler32-zlib) [ 1659.640359] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 1659.893737] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1661.536902] Key type lgssc registered [ 1662.915624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1670.138184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1670.158130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1671.450605] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1671.478280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1671.600165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.496507] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1684.772081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.835792] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1684.856591] Lustre: 57580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1685.055378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1689.727382] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1697.024816] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1702.976117] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1708.802802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1708.991967] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1709.191680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1715.229909] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1716.906949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1721.754719] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1728.447251] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1513h16m34s [ 1729.957715] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1736.825294] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1739.396665] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1739.456682] LustreError: 57587:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1759.531618] Lustre: lustre-MDT0000: Client dc29e4f1-f048-48bc-a614-f90fc7cf6caa (at 192.168.203.27@tcp) reconnecting [ 1779.511697] LustreError: 57587:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1782.054941] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1784.737489] Lustre: DEBUG MARKER: done [ 1791.797813] Lustre: server umount lustre-OST0000 complete [ 1797.600586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1797.606096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.621643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1798.689913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1798.699630] Lustre: Skipped 2 previous similar messages [ 1804.768741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1804.772173] Lustre: Skipped 1 previous similar message [ 1805.858523] Lustre: server umount lustre-MDT0000 complete [ 1809.202833] LustreError: 57571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814276 with bad export cookie 3220490953329513483 [ 1809.211618] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1809.217868] LustreError: 57571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1819.816195] Lustre: server umount lustre-MDT0001 complete [ 1829.029689] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1830.998460] Key type lgssc unregistered [ 1831.217534] LNet: 61911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.226842] LNetError: 61911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.247210] LNet: Removed LNI 192.168.203.127@tcp [ 1832.033117] Key type .llcrypt unregistered [ 1832.035865] Key type ._llcrypt unregistered [ 1874.853362] Key type ._llcrypt registered [ 1874.859153] Key type .llcrypt registered [ 1874.978679] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1876.263585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1876.302552] alg: No test for adler32 (adler32-zlib) [ 1877.404716] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 1877.628395] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1879.343488] Key type lgssc registered [ 1880.386410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1884.176953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1884.200354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.547691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1885.562696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1885.688271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1886.725268] Lustre: Failing over lustre-MDT0000 [ 1887.152731] Lustre: server umount lustre-MDT0000 complete [ 1894.136938] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:19:19 (1786814359) [ 1900.008973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.435441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.396468] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1912.817911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.934755] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1912.990839] Lustre: 64711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1913.467611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1917.628735] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1925.085135] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1930.396801] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1937.338718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1937.627615] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1937.830496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1943.194890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1943.672291] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1951.019155] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1956.969742] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1963.195396] Lustre: server umount lustre-MDT0000 complete [ 1964.009929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.021922] Lustre: Skipped 2 previous similar messages [ 1966.730336] LustreError: 64704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814433 with bad export cookie 6646995533561180994 [ 1966.744020] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1966.747630] LustreError: 64704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1967.029229] Lustre: server umount lustre-MDT0001 complete [ 1969.771587] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1973.855891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1981.625077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1985.503142] Lustre: 63637:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814436/real 1786814436] req@000000005276b7d7 x1873610637611776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1985.527985] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.615256] Lustre: 63637:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814438/real 1786814438] req@00000000cb8d6684 x1873610637611904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1990.623639] Lustre: 63635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814441/real 1786814441] req@00000000da79cb74 x1873610637612032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1991.718450] LustreError: 63633:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005fb3b23f x1873610637612928/t0(0) o250->MGC192.168.203.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1992.015375] LustreError: 66560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1992.026546] LustreError: 66560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1992.128037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.264952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1993.184512] Lustre: 63635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814443/real 1786814443] req@0000000076f3d963 x1873610637612288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814459 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1995.237848] LustreError: 66561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1995.259691] LustreError: 66561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1996.560987] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1997.280062] Lustre: 63637:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814448/real 1786814448] req@00000000acde1583 x1873610637612416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814464 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2000.364283] LustreError: 68486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2000.383322] LustreError: 68486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2004.456889] LustreError: 68486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.460388] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2004.475288] LustreError: 68486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2004.885782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.229063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2009.005622] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2010.612078] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2015.872452] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2021.948825] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2029.069623] Lustre: server umount lustre-MDT0000 complete [ 2031.081605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2031.088660] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.108463] LustreError: 66561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2031.119807] Lustre: Skipped 1 previous similar message [ 2031.148829] LustreError: 66561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2034.184141] LustreError: 64703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814501 with bad export cookie 6646995533561183269 [ 2034.215752] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2034.235900] LustreError: 64703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2034.677086] Lustre: server umount lustre-MDT0001 complete [ 2043.420968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2052.320531] Lustre: 63635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786814503/real 1786814503] req@000000004947bf21 x1873610637637888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786814519 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2052.354695] Lustre: 63635:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2052.367505] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.386126] Lustre: Skipped 1 previous similar message [ 2054.839423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2059.948126] LustreError: 66560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.071671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2060.149131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2065.863320] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2071.557695] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2076.642211] LustreError: 71171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.694461] LustreError: 71171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2076.726476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.981378] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2082.793665] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2090.743905] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2095.739164] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2102.338785] Lustre: server umount lustre-MDT0000 complete [ 2102.755269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2102.762055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.096464] LustreError: 70314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814573 with bad export cookie 6646995533561184648 [ 2106.109325] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2106.116039] LustreError: 70314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2106.501663] Lustre: server umount lustre-MDT0001 complete [ 2112.368784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2115.618211] Lustre: DEBUG MARKER: files compared the same [ 2128.118400] Lustre: server umount lustre-OST0000 complete [ 2144.978962] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2147.795063] Key type lgssc unregistered [ 2148.058951] LNet: 74447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2148.068588] LNetError: 74447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.089948] LNet: Removed LNI 192.168.203.127@tcp [ 2148.795321] Key type .llcrypt unregistered [ 2148.800474] Key type ._llcrypt unregistered [ 2174.101895] Key type ._llcrypt registered [ 2174.103691] Key type .llcrypt registered [ 2174.167479] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2176.469063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2176.532380] alg: No test for adler32 (adler32-zlib) [ 2177.807153] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 2178.437922] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2180.416103] Key type lgssc registered [ 2181.662290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2191.686658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2191.727629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.844902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2198.959142] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug +malloc all [ 2203.104794] LustreError: 75904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2203.127068] LustreError: 75904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2208.135865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2208.226940] LustreError: 75905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2208.476556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2212.766925] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug +malloc all [ 2221.205656] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2227.253269] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2233.519845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2233.728169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.832542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2239.066730] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug +malloc all [ 2246.673133] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2258.402773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.406367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2258.436259] Lustre: Skipped 1 previous similar message [ 2258.491534] Lustre: Skipped 1 previous similar message [ 2259.560746] Lustre: server umount lustre-OST0000 complete [ 2263.520991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.535802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2263.547473] Lustre: Skipped 1 previous similar message [ 2268.640381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2268.660099] Lustre: Skipped 1 previous similar message [ 2269.777912] Lustre: server umount lustre-MDT0000 complete [ 2273.775145] LustreError: 75904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.842637] LustreError: 75904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2275.405554] LustreError: 75884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814742 with bad export cookie 9842157132206636029 [ 2275.428791] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2275.433571] LustreError: 75884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2276.272813] Lustre: server umount lustre-MDT0001 complete [ 2290.058238] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2292.804089] Key type lgssc unregistered [ 2293.163640] LNet: 79439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.183221] LNetError: 79439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.200093] LNet: Removed LNI 192.168.203.127@tcp [ 2294.206081] Key type .llcrypt unregistered [ 2294.207865] Key type ._llcrypt unregistered [ 2322.700739] Key type ._llcrypt registered [ 2322.706414] Key type .llcrypt registered [ 2322.793572] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2324.198688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2324.401723] alg: No test for adler32 (adler32-zlib) [ 2325.620124] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 2326.091592] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2327.791191] Key type lgssc registered [ 2329.469267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2336.880728] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2336.897670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2352.610156] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2357.728106] LustreError: 80830:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.127@tcp: failed processing log, type 4: rc = -110 [ 2384.480496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.619232] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2397.626792] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2406.481183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.645407] Lustre: *** cfs_fail_loc=706, val=0*** [ 2407.141084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.225252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2411.759197] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2412.515508] LustreError: 80855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.529437] LustreError: 80855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2417.642303] LustreError: 80856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.682958] LustreError: 81914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.708374] LustreError: 81914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2421.091275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.614656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2426.209183] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2435.014734] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2441.355805] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2447.329872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2447.330413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2447.342960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2447.354871] Lustre: Skipped 1 previous similar message [ 2449.637876] Lustre: server umount lustre-OST0000 complete [ 2457.568345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2457.570314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.588336] Lustre: Skipped 1 previous similar message [ 2457.592443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2457.599251] Lustre: Skipped 1 previous similar message [ 2459.790905] Lustre: server umount lustre-MDT0000 complete [ 2462.688732] LustreError: 82930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2462.714298] LustreError: 82930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2463.298527] LustreError: 80838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814930 with bad export cookie 4643657656283390827 [ 2463.315561] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2463.618513] Lustre: server umount lustre-MDT0001 complete [ 2472.466106] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2475.062109] Key type lgssc unregistered [ 2475.494961] LNet: 84445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2475.505340] LNetError: 84445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2475.520830] LNet: Removed LNI 192.168.203.127@tcp [ 2476.349349] Key type .llcrypt unregistered [ 2476.354212] Key type ._llcrypt unregistered [ 2501.399391] Key type ._llcrypt registered [ 2501.401427] Key type .llcrypt registered [ 2501.516484] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2502.686751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2502.784302] alg: No test for adler32 (adler32-zlib) [ 2504.030563] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 2504.264730] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2505.951186] Key type lgssc registered [ 2506.686775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2513.871199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2513.882791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.638354] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2526.155602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2526.713251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2530.791919] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2531.810083] LustreError: 86550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.841124] LustreError: 86550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2536.928143] LustreError: 86553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.955098] LustreError: 86553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2539.226977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2539.463307] LustreError: 86554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.479348] LustreError: 86554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2539.500236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.613628] LustreError: 86553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2544.634137] LustreError: 86553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2546.007607] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2549.735378] LustreError: 86554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.761972] LustreError: 86554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2554.060094] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2554.074529] Lustre: MGS: Client 7a524e67-b0c4-4afd-89cd-9608034b1d72 (at 0@lo) reconnecting [ 2554.102217] LustreError: 86564:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2554.352702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2554.447306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2558.950142] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2574.821131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.835099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2574.836147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2577.189083] Lustre: server umount lustre-OST0000 complete [ 2585.062862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.069058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2585.072741] Lustre: Skipped 1 previous similar message [ 2585.074484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2585.099890] Lustre: Skipped 2 previous similar messages [ 2590.180445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2590.189103] Lustre: Skipped 1 previous similar message [ 2594.274118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2594.280840] Lustre: Skipped 1 previous similar message [ 2595.296309] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2595.387818] Lustre: server umount lustre-MDT0000 complete [ 2599.299117] Lustre: server umount lustre-MDT0001 complete [ 2602.497499] Lustre: server umount MGS complete [ 2612.309799] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:31:17 (1786815077) [ 2647.361098] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 2655.779381] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2693.691433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.355297] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2708.738595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.031671] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2709.069771] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2709.168390] Lustre: lustre-MDT0001: new disk, initializing [ 2709.260576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2709.287055] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2709.296771] Lustre: 93723:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2709.311486] LustreError: 93723:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2713.122165] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2723.834407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.976337] Lustre: lustre-OST0000: new disk, initializing [ 2723.982220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2723.989345] Lustre: 94489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2724.058915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2730.175195] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2730.596329] Lustre: 94784:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d11bc294 x1873611295072256/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786815213 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2740.158299] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2746.801467] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2746.812910] Lustre: MGS: Client aa318bf8-f997-44a4-91a0-e10446bbd1a9 (at 0@lo) reconnecting [ 2746.830594] LustreError: 93709:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2746.849762] Lustre: 95534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2746.885497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2746.954828] Lustre: lustre-MDT0000: new disk, initializing [ 2747.075943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2747.098384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2747.116481] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2748.192551] Lustre: 95558:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000c60d4cb2 x1873611295079424/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786815231 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2752.489698] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2753.286435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2753.310065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2753.381996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2779.104627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2779.116092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2779.132215] Lustre: Skipped 1 previous similar message [ 2779.139939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2783.725404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2783.740919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.959105] Lustre: server umount lustre-OST0000 complete [ 2788.833022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2788.834393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.852445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2788.890182] Lustre: Skipped 1 previous similar message [ 2788.893541] Lustre: Skipped 2 previous similar messages [ 2803.167637] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2803.324557] Lustre: server umount lustre-MDT0000 complete [ 2813.271201] Lustre: server umount lustre-MDT0001 complete [ 2817.056888] Lustre: server umount MGS complete [ 2827.302758] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:34:52 (1786815292) [ 2843.414569] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2846.671094] Key type lgssc unregistered [ 2846.921988] LNet: 98061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2846.937113] LNetError: 98061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2846.988363] LNet: Removed LNI 192.168.203.127@tcp [ 2847.989989] Key type .llcrypt unregistered [ 2847.992448] Key type ._llcrypt unregistered [ 2863.622285] Key type ._llcrypt registered [ 2863.628168] Key type .llcrypt registered [ 2863.750144] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2864.927536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2864.941204] alg: No test for adler32 (adler32-zlib) [ 2865.985644] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 2866.275840] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2867.943274] Key type lgssc registered [ 2869.012918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2876.212795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2876.228644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.756184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2881.894511] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2887.137860] LustreError: 99262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.149351] LustreError: 99262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2890.179378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.457197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2895.104697] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2902.281209] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2908.115924] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2914.690360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2914.976577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.341979] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2922.203804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2928.634421] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2936.801178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2936.829433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2936.869681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2937.824695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2937.834377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2937.844392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.907121] Lustre: Skipped 1 previous similar message [ 2942.953944] LustreError: 101842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2943.005604] LustreError: 101842:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2943.420931] Lustre: server umount lustre-OST0000 complete [ 2948.067321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2948.081617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2948.090842] Lustre: Skipped 1 previous similar message [ 2953.185440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.190091] Lustre: Skipped 1 previous similar message [ 2953.826785] Lustre: server umount lustre-MDT0000 complete [ 2957.441067] LustreError: 99245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815424 with bad export cookie 14570099553556055076 [ 2957.446358] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2957.450384] LustreError: 99245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2957.764268] Lustre: server umount lustre-MDT0001 complete [ 2968.179628] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2971.069248] Key type lgssc unregistered [ 2971.415804] LNet: 102800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2971.425083] LNetError: 102800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2971.456683] LNet: Removed LNI 192.168.203.127@tcp [ 2972.261985] Key type .llcrypt unregistered [ 2972.267798] Key type ._llcrypt unregistered [ 2987.401696] Key type ._llcrypt registered [ 2987.404678] Key type .llcrypt registered [ 2987.550981] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2988.488180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2988.554806] alg: No test for adler32 (adler32-zlib) [ 2989.756780] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 2990.106216] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2991.776262] Key type lgssc registered [ 2993.076157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2998.767780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2998.781862] LustreError: 103880:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2998.803811] LustreError: 103883:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2998.808284] LustreError: 103880:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 2998.813609] LustreError: 103880:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2998.823028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.825602] LustreError: 103880:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3000.380371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.401794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3007.631630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3011.575873] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3016.199704] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing lsmod [ 3018.984955] LustreError: 105089:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3018.995971] LustreError: 105088:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3019.004059] LustreError: 105089:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3019.013868] LustreError: 105089:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3019.023631] LustreError: 105089:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3019.032419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3019.410019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.330087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3023.343836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3023.367654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3024.876773] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3024.881978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3024.889795] Lustre: Skipped 1 previous similar message [ 3024.907854] Lustre: Skipped 2 previous similar messages [ 3028.485959] Lustre: server umount lustre-MDT0000 complete [ 3029.985022] LustreError: 103921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.000459] LustreError: 103921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3031.805900] LustreError: 103899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815498 with bad export cookie 3025688850716428109 [ 3031.811410] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3031.826077] LustreError: 103899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3032.101543] Lustre: server umount lustre-MDT0001 complete [ 3045.793242] Lustre: server umount lustre-OST0000 complete [ 3052.261282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.720871] LustreError: 106153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.824714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3057.048435] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3058.159263] LustreError: 106154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.252874] LustreError: 106153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.363958] LustreError: 106154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3068.050395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.592826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3073.488435] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3079.735843] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3085.121931] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3091.612693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3091.917895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.275641] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3098.107670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3104.070308] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3113.443242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.446727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3113.457765] Lustre: Skipped 1 previous similar message [ 3113.458173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3113.469758] Lustre: Skipped 1 previous similar message [ 3116.095539] Lustre: server umount lustre-OST0000 complete [ 3119.584433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3119.590691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3119.598465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3123.683358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3123.683989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3124.859969] Lustre: server umount lustre-MDT0000 complete [ 3127.655590] LustreError: 106887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815594 with bad export cookie 3025688850716428795 [ 3127.659026] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3127.665514] LustreError: 106887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3127.919038] Lustre: server umount lustre-MDT0001 complete [ 3137.322827] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3139.770857] Key type lgssc unregistered [ 3140.042376] LNet: 109686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3140.047961] LNetError: 109686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3140.063556] LNet: Removed LNI 192.168.203.127@tcp [ 3140.819882] Key type .llcrypt unregistered [ 3140.824442] Key type ._llcrypt unregistered [ 3161.837943] Key type ._llcrypt registered [ 3161.840582] Key type .llcrypt registered [ 3161.967470] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3163.227731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3163.243253] alg: No test for adler32 (adler32-zlib) [ 3164.257661] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 3164.442287] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3166.103188] Key type lgssc registered [ 3167.163770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3174.507392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3174.517683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3176.031750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.507633] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3185.126286] LustreError: 111104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.148988] LustreError: 111104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3189.379957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3189.652857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3194.149912] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3200.857289] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3205.621216] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3211.771760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3212.041191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.393233] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3220.286258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3224.179677] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3228.817175] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3231.328450] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3235.811538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3235.816233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3235.832208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3235.846517] Lustre: Skipped 1 previous similar message [ 3237.170488] Lustre: server umount lustre-OST0000 complete [ 3240.928344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3240.931857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.934388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3240.956915] Lustre: Skipped 2 previous similar messages [ 3246.056118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3246.068915] Lustre: Skipped 2 previous similar messages [ 3246.984393] Lustre: server umount lustre-MDT0000 complete [ 3250.213201] LustreError: 111085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815717 with bad export cookie 5553447577154651663 [ 3250.236353] LustreError: 111085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3250.236443] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3250.600785] Lustre: server umount lustre-MDT0001 complete [ 3259.413040] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3262.356858] Key type lgssc unregistered [ 3262.639613] LNet: 114741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3262.647127] LNetError: 114741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3262.672623] LNet: Removed LNI 192.168.203.127@tcp [ 3263.480692] Key type .llcrypt unregistered [ 3263.482118] Key type ._llcrypt unregistered [ 3279.628298] Key type ._llcrypt registered [ 3279.635099] Key type .llcrypt registered [ 3279.738251] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3280.573982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3280.791489] alg: No test for adler32 (adler32-zlib) [ 3281.825120] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 3282.027687] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3283.687179] Key type lgssc registered [ 3284.752711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3291.298802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3291.313192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3292.767477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3296.686700] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3302.368845] LustreError: 115977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3302.385736] LustreError: 115977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3303.797993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3304.086365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3308.197947] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3316.240417] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3320.847442] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3327.596951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3328.021884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.224782] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3337.269844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3341.121132] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3349.472451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3349.480997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.489659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3351.020316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3351.030348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3351.052853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3352.548106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3355.206692] Lustre: server umount lustre-OST0000 complete [ 3360.224888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3360.232203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3360.247756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3360.250521] Lustre: Skipped 1 previous similar message [ 3362.792118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3365.084368] Lustre: server umount lustre-MDT0000 complete [ 3367.905408] LustreError: 115977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3367.924948] LustreError: 115977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3368.217486] LustreError: 117821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815835 with bad export cookie 10015038661387147803 [ 3368.226529] LustreError: 117821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3368.226812] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3368.547659] Lustre: server umount lustre-MDT0001 complete [ 3379.167282] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3382.206062] Key type lgssc unregistered [ 3382.554780] LNet: 119512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3382.564525] LNetError: 119512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3382.589428] LNet: Removed LNI 192.168.203.127@tcp [ 3383.551537] Key type .llcrypt unregistered [ 3383.555961] Key type ._llcrypt unregistered [ 3410.889268] Key type ._llcrypt registered [ 3410.892811] Key type .llcrypt registered [ 3411.020971] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3413.067976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3413.117740] alg: No test for adler32 (adler32-zlib) [ 3414.366975] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 3414.741278] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3416.439182] Key type lgssc registered [ 3417.672935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3426.275961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3426.296057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.611537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3432.930429] LustreError: 120928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.982104] LustreError: 120928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3434.330706] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3438.049613] LustreError: 120929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3442.694443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.036412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3448.413706] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3457.312939] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3463.579663] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3470.384520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3470.728941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.830490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3476.483865] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3483.954515] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3495.028250] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3498.470417] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3498.476888] Lustre: Skipped 2 previous similar messages [ 3502.775983] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3502.780828] Lustre: Skipped 2 previous similar messages [ 3503.140209] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3503.156612] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3513.368068] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.27@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3513.388687] Lustre: Skipped 2 previous similar messages [ 3518.570661] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.27@tcp 192.168.0.[2,10]@tcp [ 3522.897914] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.27/24@tcp in log lustre-MDT0000 [ 3522.910095] Lustre: Skipped 2 previous similar messages [ 3525.629221] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.27/24@tcp [ 3525.648152] Lustre: Skipped 1 previous similar message [ 3536.893448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3536.897973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.902646] LustreError: Skipped 1 previous similar message [ 3536.924966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3537.572863] Lustre: server umount lustre-OST0000 complete [ 3543.521524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3543.541735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3543.545484] Lustre: Skipped 1 previous similar message [ 3547.519483] Lustre: server umount lustre-MDT0000 complete [ 3548.643157] LustreError: 122363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3548.685145] LustreError: 122363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3550.992964] LustreError: 123559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816017 with bad export cookie 612889661191089162 [ 3551.000897] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3551.017946] LustreError: 123559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3560.654923] Lustre: server umount lustre-MDT0001 complete [ 3570.798405] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3574.393595] Key type lgssc unregistered [ 3574.693787] LNet: 125726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3574.707293] LNetError: 125726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3574.735327] LNet: Removed LNI 192.168.203.127@tcp [ 3575.727479] Key type .llcrypt unregistered [ 3575.732094] Key type ._llcrypt unregistered [ 3612.585660] Key type ._llcrypt registered [ 3612.588367] Key type .llcrypt registered [ 3612.677918] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3613.564023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3613.733050] alg: No test for adler32 (adler32-zlib) [ 3614.821078] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 3615.230867] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3616.927545] Key type lgssc registered [ 3618.282386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3625.797609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3625.821448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.326103] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3627.348570] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.27/24@tcp [ 3627.404054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3631.866467] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3637.738167] LustreError: 127290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3637.763285] LustreError: 127290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3640.233834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.635456] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3640.673330] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.27/24@tcp [ 3640.713578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3645.111131] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3652.979120] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3658.976967] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3666.056782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3666.329929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.323863] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3673.527884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3679.795671] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3693.042624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3693.053606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3693.067823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.100762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3696.102113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.110740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3696.120450] Lustre: Skipped 1 previous similar message [ 3701.217556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.335131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3706.339841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.369874] Lustre: Skipped 2 previous similar messages [ 3706.797683] Lustre: server umount lustre-OST0000 complete [ 3716.577966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3716.586159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3716.608309] Lustre: Skipped 1 previous similar message [ 3716.631405] Lustre: Skipped 1 previous similar message [ 3720.894189] Lustre: server umount lustre-MDT0000 complete [ 3721.696138] LustreError: 127285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.733440] LustreError: 127285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3726.266034] LustreError: 127272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816193 with bad export cookie 9699151886242733083 [ 3726.268635] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3726.271696] LustreError: 127272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3726.606501] Lustre: server umount lustre-MDT0001 complete [ 3738.585311] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3740.969653] Key type lgssc unregistered [ 3741.242647] LNet: 130892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3741.248858] LNetError: 130892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3741.260274] LNet: Removed LNI 192.168.203.127@tcp [ 3742.043265] Key type .llcrypt unregistered [ 3742.046393] Key type ._llcrypt unregistered [ 3768.789575] Key type ._llcrypt registered [ 3768.792424] Key type .llcrypt registered [ 3768.890930] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 3784.656554] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3785.735984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3785.826101] alg: No test for adler32 (adler32-zlib) [ 3786.967651] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 3787.235656] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3788.903195] Key type lgssc registered [ 3790.001938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3825.651751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3825.696248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3827.010151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3827.043784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3827.140536] Lustre: lustre-MDT0000: new disk, initializing [ 3827.241436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3827.256382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3832.555106] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3844.288963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3844.453798] Lustre: 134293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3844.541717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3844.548243] Lustre: Skipped 1 previous similar message [ 3844.696720] Lustre: lustre-MDT0001: new disk, initializing [ 3844.826537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3844.862447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3844.892519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3849.706829] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3859.060911] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3864.841947] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3871.789072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3872.054621] Lustre: lustre-OST0000: new disk, initializing [ 3872.057973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3872.064330] Lustre: 136265:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3872.150813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3877.965543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3877.978595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3878.097182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3878.803339] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3887.520342] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3892.263530] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3892.428584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3896.183512] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3896.367215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3898.848029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3898.861366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3898.864663] LustreError: Skipped 1 previous similar message [ 3898.866671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3898.877991] Lustre: Skipped 1 previous similar message [ 3898.901692] Lustre: Skipped 1 previous similar message [ 3903.972754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3903.976772] Lustre: Skipped 1 previous similar message [ 3904.119058] Lustre: server umount lustre-OST0000 complete [ 3909.090634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3909.113846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3909.127332] Lustre: Skipped 1 previous similar message [ 3914.210573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3914.222069] Lustre: Skipped 1 previous similar message [ 3919.329451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3921.375159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3923.602357] Lustre: server umount lustre-MDT0000 complete [ 3927.710263] LustreError: 134287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816394 with bad export cookie 1305060308811660368 [ 3927.713897] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3927.721320] LustreError: 134287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3928.011707] Lustre: server umount lustre-MDT0001 complete [ 3937.112655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3937.683554] LustreError: 138483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3937.712129] LustreError: 138483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3937.771900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3942.390108] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3942.885763] LustreError: 138484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3948.009568] LustreError: 138483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3951.011618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3951.244569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3955.945316] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3965.013810] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3972.394637] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3980.094195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3980.593157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3987.411253] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3995.275445] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4000.238784] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4000.459680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4004.283299] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4004.516082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4013.890833] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4014.047064] Lustre: lustre-OST0001: new disk, initializing [ 4014.051476] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4014.066533] Lustre: 141855:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4014.240066] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4019.928286] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4022.355482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4022.371342] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4022.454128] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4028.992790] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4029.243604] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4035.521574] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4035.710400] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4042.032390] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff956e5255e800.ost_server_uuid 50 [ 4043.963287] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff956e5255e800.ost_server_uuid in IDLE state after 0 sec [ 4053.473542] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4053.479953] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4053.486373] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4053.488586] Lustre: Skipped 2 previous similar messages [ 4053.500110] Lustre: Skipped 2 previous similar messages [ 4058.760269] Lustre: server umount lustre-OST0001 complete [ 4063.722379] LustreError: 140306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4063.723815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4063.726809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4063.726817] LustreError: Skipped 1 previous similar message [ 4063.741567] LustreError: 140306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4063.772871] Lustre: Skipped 1 previous similar message [ 4068.841888] LustreError: 143246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.860361] LustreError: 143246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4069.191525] Lustre: server umount lustre-OST0000 complete [ 4073.953100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4073.964486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4073.985143] Lustre: Skipped 1 previous similar message [ 4074.015416] Lustre: Skipped 7 previous similar messages [ 4078.957914] Lustre: server umount lustre-MDT0000 complete [ 4079.074682] LustreError: 138484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4079.103182] LustreError: 138484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4084.023855] LustreError: 140326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816550 with bad export cookie 1305060308811661250 [ 4084.037478] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4084.043068] LustreError: 140326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4084.393437] Lustre: server umount lustre-MDT0001 complete [ 4106.000482] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 4108.339517] Key type lgssc unregistered [ 4108.676788] LNet: 144861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4108.686812] LNetError: 144861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4108.710759] LNet: Removed LNI 192.168.203.127@tcp [ 4109.586815] Key type .llcrypt unregistered [ 4109.590413] Key type ._llcrypt unregistered [ 4149.142252] Key type ._llcrypt registered [ 4149.143958] Key type .llcrypt registered [ 4149.239104] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 4163.574288] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4164.657335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4164.805403] alg: No test for adler32 (adler32-zlib) [ 4165.803615] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 4166.095274] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 4167.823171] Key type lgssc registered [ 4168.940143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4209.264981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4209.303993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4210.452604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4210.506531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4210.585294] Lustre: lustre-MDT0000: new disk, initializing [ 4210.682343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4210.705801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4215.405055] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4227.441324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.671282] Lustre: 149016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4227.767150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4227.778641] Lustre: Skipped 1 previous similar message [ 4227.891899] Lustre: lustre-MDT0001: new disk, initializing [ 4228.029562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4228.070189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4228.088567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4232.281236] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4241.356511] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4247.019593] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4253.832897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4254.145575] Lustre: lustre-OST0000: new disk, initializing [ 4254.148887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4254.157956] Lustre: 150988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4254.234384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4260.492316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4260.500965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4260.650660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4261.411617] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4271.822575] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4276.866252] Lustre: Failing over lustre-OST0000 [ 4276.987925] Lustre: server umount lustre-OST0000 complete [ 4294.628651] Lustre: 147241:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786816745/real 1786816745] req@0000000087a0afee x1873613037799552/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786816761 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4294.649479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.896497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4295.184542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4295.234049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4295.247131] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4296.006414] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4297.695168] Lustre: 147239:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786816748/real 1786816748] req@00000000ec6d9126 x1873613037800192/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786816764 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4301.151963] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4302.822292] Lustre: 147238:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786816753/real 1786816753] req@000000001abc9da3 x1873613037801856/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786816769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4302.861617] Lustre: 147238:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4304.401868] Lustre: lustre-OST0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [ 4304.405154] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4304.801256] Lustre: Failing over lustre-MDT0000 [ 4304.865584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4304.879584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4304.902087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4305.077869] Lustre: server umount lustre-MDT0000 complete [ 4307.871151] Lustre: 147239:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786816758/real 1786816758] req@000000001cd5b7b0 x1873613037802880/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786816774 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4307.882199] Lustre: 147239:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4309.475276] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4309.488672] Lustre: Skipped 1 previous similar message [ 4311.310177] LustreError: 149022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4311.328942] LustreError: 149022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4314.593424] LustreError: 149023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4314.615410] LustreError: 149023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4316.433114] LustreError: 150739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4319.726830] LustreError: 150739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4319.752499] LustreError: 150739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4322.861059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4322.932385] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4323.135165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4323.217301] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4327.106138] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4327.691455] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4328.435304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4328.442193] Lustre: Skipped 1 previous similar message [ 4328.486046] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4328.563779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4338.657803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.668299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4338.670199] Lustre: Skipped 1 previous similar message [ 4338.679085] Lustre: Skipped 1 previous similar message [ 4343.063688] Lustre: server umount lustre-OST0000 complete [ 4348.896590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.942103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4348.956795] Lustre: Skipped 1 previous similar message [ 4352.880731] Lustre: server umount lustre-MDT0000 complete [ 4354.016945] LustreError: 149027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4354.057749] LustreError: 149027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4356.264690] LustreError: 149009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816823 with bad export cookie 14542774399130421213 [ 4356.284796] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4356.287127] LustreError: 149009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4356.695665] Lustre: server umount lustre-MDT0001 complete [ 4366.106966] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 4368.469144] Key type lgssc unregistered [ 4368.777602] LNet: 154647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4368.785928] LNetError: 154647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4368.800208] LNet: Removed LNI 192.168.203.127@tcp [ 4369.695254] Key type .llcrypt unregistered [ 4369.699102] Key type ._llcrypt unregistered [ 4405.366558] Key type ._llcrypt registered [ 4405.369571] Key type .llcrypt registered [ 4405.482335] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4406.583171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4406.609672] alg: No test for adler32 (adler32-zlib) [ 4407.653639] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 4407.923190] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 4409.568312] Key type lgssc registered [ 4410.521880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4417.277682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4417.291028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.716516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4422.745994] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4428.258339] LustreError: 156177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4428.287512] LustreError: 156177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4432.063219] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4433.379482] LustreError: 156176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4436.000518] LustreError: 156205:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4436.013616] LustreError: 156205:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4436.021051] LustreError: 156205:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4436.292163] Lustre: server umount lustre-MDT0000 complete [ 4449.591583] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 4458.963993] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4505.099846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.370491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4505.396567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4505.456902] Lustre: lustre-MDT0000: new disk, initializing [ 4505.541064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4505.569184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4513.063587] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4525.876421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4526.026177] Lustre: 159507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4526.083243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4526.087482] Lustre: Skipped 1 previous similar message [ 4526.152917] Lustre: lustre-MDT0001: new disk, initializing [ 4526.240781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4526.283589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4526.300314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4531.220846] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4541.961565] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4548.385792] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4555.395047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4555.647650] Lustre: lustre-OST0000: new disk, initializing [ 4555.654212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4555.661233] Lustre: 161478:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4555.741386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4561.469492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4561.491958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4561.727520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4562.811444] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4573.473751] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4578.812574] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4581.610339] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:04:06 (1786817046) [ 4853.729189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4853.730648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4853.749059] Lustre: Skipped 1 previous similar message [ 4853.782970] Lustre: Skipped 2 previous similar messages [ 4857.169816] Lustre: server umount lustre-MDT0000 complete [ 4858.854634] LustreError: 161015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4858.878222] LustreError: 161015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4861.725341] LustreError: 159500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817328 with bad export cookie 1885951381941030502 [ 4861.730162] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4861.743933] LustreError: 159500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4863.967628] LustreError: 159516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.976340] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4864.084125] Lustre: Skipped 1 previous similar message [ 4864.087848] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4866.989985] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4871.073884] LustreError: 159514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4871.073979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4871.110455] LustreError: 159514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4876.255132] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4876.258426] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4876.637775] Lustre: server umount lustre-MDT0001 complete [ 4888.253944] Lustre: server umount lustre-OST0000 complete [ 4898.695246] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 4908.897780] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4951.750065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.053866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4952.087390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4952.228692] Lustre: lustre-MDT0000: new disk, initializing [ 4952.331173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4952.348536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4957.011835] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4968.420741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4968.619713] Lustre: 166167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4968.702549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4968.706648] Lustre: Skipped 1 previous similar message [ 4968.810892] Lustre: lustre-MDT0001: new disk, initializing [ 4968.894645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4968.930848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4968.938309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4973.673917] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4983.309803] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4989.295390] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4996.111586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4996.482698] Lustre: lustre-OST0000: new disk, initializing [ 4996.487849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4996.496334] Lustre: 168135:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4996.651484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5002.858916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5002.875139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5002.994941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5003.024347] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5011.998416] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5017.192873] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5017.435782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5021.242687] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5021.399268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5023.713699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5023.716579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5023.726069] LustreError: Skipped 1 previous similar message [ 5023.729089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5023.750663] Lustre: Skipped 1 previous similar message [ 5023.770077] Lustre: Skipped 1 previous similar message [ 5029.363111] Lustre: server umount lustre-OST0000 complete [ 5033.957697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5033.980745] Lustre: Skipped 1 previous similar message [ 5038.982755] Lustre: server umount lustre-MDT0000 complete [ 5039.072763] LustreError: 166178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5039.091650] LustreError: 166178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5042.804399] LustreError: 166961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817509 with bad export cookie 1885951381941189332 [ 5042.814261] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5042.815719] LustreError: 166961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5043.019493] Lustre: server umount lustre-MDT0001 complete [ 5051.047411] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:11:56 (1786817516) [ 5065.710658] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 5074.307871] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5110.357786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5110.610570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5110.633538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5110.698598] Lustre: lustre-MDT0000: new disk, initializing [ 5110.771540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5110.787140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5115.360585] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5125.411477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5125.570281] Lustre: 172891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5125.847710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5129.955907] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5138.585211] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5144.096772] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5150.099699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5150.323590] Lustre: lustre-OST0000: new disk, initializing [ 5150.326967] Lustre: Skipped 1 previous similar message [ 5150.330833] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5150.336337] Lustre: Skipped 2 previous similar messages [ 5150.345566] Lustre: 174865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5155.971513] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5156.954508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5156.967936] Lustre: Skipped 1 previous similar message [ 5156.979645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5157.080515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5164.742249] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5172.716885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5172.718841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5172.729233] LustreError: Skipped 1 previous similar message [ 5172.731817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5172.731825] Lustre: Skipped 4 previous similar messages [ 5172.792974] Lustre: Skipped 1 previous similar message [ 5178.521353] Lustre: server umount lustre-OST0000 complete [ 5182.432621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5182.444533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5196.255262] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5196.443039] Lustre: server umount lustre-MDT0000 complete [ 5198.307735] LustreError: 172901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5199.699084] LustreError: 172881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817666 with bad export cookie 1885951381941190214 [ 5199.704247] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5199.706641] LustreError: 172881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5200.047725] Lustre: server umount lustre-MDT0001 complete [ 5208.426977] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:14:33 (1786817673) [ 5224.903875] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 5236.153561] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5269.956293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.156700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5270.198225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5270.327288] Lustre: lustre-MDT0000: new disk, initializing [ 5270.383121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5270.390265] Lustre: Skipped 2 previous similar messages [ 5270.413224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5274.628357] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5286.301034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5286.499058] Lustre: 179191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5286.805835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5291.070073] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5300.044233] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5305.827449] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5312.357312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5312.716330] Lustre: 181166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5319.063277] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5321.790473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5321.929399] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5328.830961] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5337.574409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5337.583494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5337.603030] Lustre: Skipped 2 previous similar messages [ 5337.605497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5337.614869] Lustre: Skipped 9 previous similar messages [ 5340.995314] Lustre: server umount lustre-OST0000 complete [ 5351.398080] Lustre: server umount lustre-MDT0000 complete [ 5352.936979] LustreError: 179201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5352.965246] LustreError: 179201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5356.703538] LustreError: 179182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817823 with bad export cookie 1885951381941191299 [ 5356.708477] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5356.736154] LustreError: 179182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5357.440319] Lustre: server umount lustre-MDT0001 complete [ 5372.640184] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5375.955554] Key type lgssc unregistered [ 5376.229750] LNet: 183025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5376.233965] LNetError: 183025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5376.255866] LNet: Removed LNI 192.168.203.127@tcp [ 5377.002691] Key type .llcrypt unregistered [ 5377.018371] Key type ._llcrypt unregistered [ 5404.476159] Key type ._llcrypt registered [ 5404.478728] Key type .llcrypt registered [ 5404.629589] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5405.784159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5405.947045] alg: No test for adler32 (adler32-zlib) [ 5407.092782] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 5407.412449] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5409.447208] Key type lgssc registered [ 5410.932227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5418.622713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5418.645427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5420.071520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5424.768040] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5429.223598] LustreError: 184441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5429.253311] LustreError: 184441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5433.135430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5433.400423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5439.402678] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5447.052024] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5453.729140] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5460.462951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5460.765381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.893936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5466.351408] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5473.909224] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5495.779231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5495.787925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5495.812510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5498.858155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5498.861298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5498.875669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5500.551298] Lustre: server umount lustre-OST0000 complete [ 5504.993711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5505.002772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5505.021547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5505.028544] Lustre: Skipped 1 previous similar message [ 5509.090490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5509.092956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5510.930551] Lustre: server umount lustre-MDT0000 complete [ 5514.209820] LustreError: 184442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5514.237471] LustreError: 184442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5514.355543] LustreError: 185176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817981 with bad export cookie 11406365035036809826 [ 5514.375613] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5514.734933] Lustre: server umount lustre-MDT0001 complete [ 5526.982752] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5530.479095] Key type lgssc unregistered [ 5530.791420] LNet: 187999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5530.802807] LNetError: 187999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5530.826429] LNet: Removed LNI 192.168.203.127@tcp [ 5532.562831] Key type .llcrypt unregistered [ 5532.567841] Key type ._llcrypt unregistered [ 5563.875298] Key type ._llcrypt registered [ 5563.879598] Key type .llcrypt registered [ 5564.015212] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5565.235458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5565.420346] alg: No test for adler32 (adler32-zlib) [ 5566.445864] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 5566.809615] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5568.463217] Key type lgssc registered [ 5569.988663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5579.409986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5579.436378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5580.932884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.726763] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5590.502798] LustreError: 189432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5590.526979] LustreError: 189432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5594.717994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5595.091386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5599.740404] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5607.822230] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5613.022039] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5618.902492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5619.298489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5625.282695] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5625.461557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5632.191191] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5638.959412] Lustre: lustre-OST0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 5641.187157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5641.191465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5641.213401] LustreError: Skipped 1 previous similar message [ 5641.217550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5641.232354] Lustre: Skipped 1 previous similar message [ 5641.269029] Lustre: Skipped 1 previous similar message [ 5644.051990] Lustre: lustre-OST0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 5645.169590] Lustre: server umount lustre-OST0000 complete [ 5654.131679] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff956e5829e000.ost_server_uuid 50 [ 5656.212110] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5671.909202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5671.914056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5671.923853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5671.967611] Lustre: Skipped 1 previous similar message [ 5676.539223] Lustre: server umount lustre-MDT0000 complete [ 5677.026673] LustreError: 189427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5677.045351] LustreError: 189427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5680.404344] LustreError: 192476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818147 with bad export cookie 5528055222875794513 [ 5680.408444] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5680.412405] LustreError: 192476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5680.768194] Lustre: server umount lustre-MDT0001 complete [ 5688.643405] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:22:34 (1786818154) [ 5695.299635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5695.612439] LustreError: 193345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.680932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5699.748883] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5701.090556] LustreError: 193346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5705.186862] LustreError: 193345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5707.442232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5707.771421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5713.103433] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5720.928555] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5726.909782] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5733.556406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5733.824800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5739.537078] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5740.976187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5746.357929] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5752.974623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5753.146532] Lustre: lustre-OST0001: new disk, initializing [ 5753.151139] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5753.161890] Lustre: 196184:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5753.294901] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5758.993775] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5761.638064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5761.652973] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5761.787122] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5767.544438] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5771.745583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5771.758531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.796474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5771.812333] Lustre: Skipped 1 previous similar message [ 5775.728707] Lustre: server umount lustre-OST0000 complete [ 5777.382808] LustreError: 195167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.405459] LustreError: 195167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5780.977216] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5781.203138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5784.737103] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5784.918589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5793.552132] LustreError: 195168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5793.582177] LustreError: 195168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5802.978499] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5802.983674] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.995382] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5802.995390] Lustre: Skipped 3 previous similar messages [ 5803.039368] Lustre: Skipped 2 previous similar messages [ 5804.135760] Lustre: server umount lustre-OST0001 complete [ 5808.098741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5808.111470] Lustre: Skipped 1 previous similar message [ 5813.914282] Lustre: server umount lustre-MDT0000 complete [ 5817.679085] LustreError: 193326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818284 with bad export cookie 5528055222875795577 [ 5817.680895] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5817.701927] LustreError: 193326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5818.306365] Lustre: server umount lustre-MDT0001 complete [ 5843.233874] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:25:08 (1786818308) [ 5849.183848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.518339] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5849.546664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5849.788610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5853.542405] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5861.418543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5861.526589] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5861.557074] Lustre: 199620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5865.698858] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5872.803868] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5877.933568] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5883.527590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5883.737763] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5889.387813] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5890.267543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5898.403643] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5907.601058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5907.656096] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5914.048462] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5920.211923] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5924.840614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.847558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5924.861182] Lustre: Skipped 1 previous similar message [ 5924.877985] Lustre: Skipped 6 previous similar messages [ 5928.669642] Lustre: server umount lustre-OST0000 complete [ 5929.966310] LustreError: 202874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.987720] LustreError: 202874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5944.804386] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5944.815825] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5948.458140] Lustre: server umount lustre-OST0001 complete [ 5954.020422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5960.673321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5960.688164] Lustre: Skipped 8 previous similar messages [ 5966.303255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5966.533954] Lustre: server umount lustre-MDT0000 complete [ 5969.730561] LustreError: 199615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818436 with bad export cookie 5528055222875796991 [ 5969.735674] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5969.748866] LustreError: 199615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5970.053016] Lustre: server umount lustre-MDT0001 complete [ 5992.099578] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:27:37 (1786818457) [ 6008.132165] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 6017.226690] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 6051.003461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6051.159796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6051.188158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6051.300266] Lustre: lustre-MDT0000: new disk, initializing [ 6051.377151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6051.389944] Lustre: Skipped 3 previous similar messages [ 6051.410987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6055.305881] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6064.434062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6064.565917] Lustre: 207720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6064.671918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6064.682380] Lustre: Skipped 1 previous similar message [ 6064.800446] Lustre: lustre-MDT0001: new disk, initializing [ 6064.958289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6064.973993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6068.892199] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6077.332791] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6082.092235] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6087.880793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6088.073303] Lustre: lustre-OST0000: new disk, initializing [ 6088.077227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6088.083621] Lustre: 209688:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6093.950532] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6095.915081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6095.925911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6096.066487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6102.258653] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6106.309467] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6106.595963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6111.039957] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6111.279458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6116.838785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6116.841021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6116.856888] Lustre: Skipped 3 previous similar messages [ 6116.861672] LustreError: Skipped 1 previous similar message [ 6116.862936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6116.878714] Lustre: Skipped 3 previous similar messages [ 6119.369746] Lustre: server umount lustre-OST0000 complete [ 6128.968334] Lustre: server umount lustre-MDT0000 complete [ 6132.195073] LustreError: 208850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6132.218865] LustreError: 208850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6132.321223] LustreError: 207711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818599 with bad export cookie 5528055222875798552 [ 6132.332336] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6132.333433] LustreError: 207711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6132.613612] Lustre: server umount lustre-MDT0001 complete [ 6140.830854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.975884] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6153.316988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6157.446933] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6164.851506] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6170.180675] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6177.027705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6185.381551] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6196.959557] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6201.793725] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6201.966577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6207.637767] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6207.857667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6212.581697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6212.590744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6212.597346] Lustre: Skipped 2 previous similar messages [ 6212.630893] LustreError: Skipped 1 previous similar message [ 6224.351133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6224.608255] Lustre: server umount lustre-OST0000 complete [ 6231.527469] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6237.672771] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6266.966154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6267.343709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6267.352232] Lustre: Skipped 5 previous similar messages [ 6272.564493] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6277.615611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.653798] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.656073] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6277.689690] Lustre: Skipped 1 previous similar message [ 6279.221881] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6285.940664] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 6290.783265] Lustre: server umount lustre-OST0000 complete [ 6297.056858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6297.064997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.070490] Lustre: Skipped 9 previous similar messages [ 6308.319423] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6308.320202] Lustre: 188743:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786818759/real 1786818759] req@00000000f19cf156 x1873614506019968/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786818775 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6308.493937] Lustre: server umount lustre-MDT0000 complete [ 6311.527698] LustreError: 211886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818778 with bad export cookie 5528055222875799427 [ 6311.530458] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6311.536732] LustreError: 211886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6311.849939] Lustre: server umount lustre-MDT0001 complete [ 6319.272192] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:33:05 (1786818785) [ 6324.954575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6325.232746] LustreError: 217871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.240314] LustreError: 217871:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6329.325391] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6338.570157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6343.090740] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6350.943904] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6356.785542] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6363.678200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6370.470996] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6377.529029] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6382.518471] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6382.826729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6387.597993] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6387.878231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6394.629045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6394.846389] Lustre: lustre-OST0001: new disk, initializing [ 6394.849743] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6394.860537] Lustre: 221241:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6401.424146] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6402.100675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6402.106612] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6402.215177] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6410.776152] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6415.281814] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6415.513296] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6419.796757] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6419.971454] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6423.016601] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6423.028798] Lustre: Skipped 4 previous similar messages [ 6427.871430] Lustre: server umount lustre-OST0001 complete [ 6434.855805] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6435.099372] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6439.340467] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6439.548077] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6468.701488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6474.235672] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6474.262691] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6474.266627] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6474.286534] Lustre: Skipped 1 previous similar message [ 6476.203283] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6484.277420] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6485.480466] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6489.568703] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6489.580206] LustreError: Skipped 2 previous similar messages [ 6493.448499] Lustre: server umount lustre-OST0001 complete [ 6515.338782] Lustre: server umount lustre-OST0000 complete [ 6525.386350] Lustre: server umount lustre-MDT0000 complete [ 6528.963941] LustreError: 221240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786818995 with bad export cookie 5528055222875800400 [ 6528.974720] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6528.978228] LustreError: 221240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6543.327309] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6543.522058] Lustre: server umount lustre-MDT0001 complete [ 6566.209383] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:37:11 (1786819031) [ 6573.022406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6573.433267] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6573.484727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6573.715465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6573.730754] Lustre: Skipped 5 previous similar messages [ 6577.952168] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6585.588736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6585.737413] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6585.770359] Lustre: 226489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6590.594242] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6598.655460] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6607.512873] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6617.289442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6617.525844] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6622.580548] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6629.711303] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6638.999189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6639.052821] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6645.229092] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6651.813744] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6653.323419] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6657.672803] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6657.980201] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6662.971503] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6663.211771] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6668.040262] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff956e47931000.ost_server_uuid 50 [ 6669.651583] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff956e47931000.ost_server_uuid in IDLE state after 0 sec [ 6670.734766] Lustre: Permanently deactivating lustre-OST0001 [ 6670.775365] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6674.768462] Lustre: Permanently reactivating lustre-OST0001 [ 6674.799165] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6674.811866] Lustre: Skipped 2 previous similar messages [ 6680.076615] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.080793] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6680.111395] Lustre: Skipped 8 previous similar messages [ 6680.129890] Lustre: Skipped 17 previous similar messages [ 6694.367556] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6694.481345] Lustre: server umount lustre-OST0001 complete [ 6695.403782] LustreError: 228334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6695.424559] LustreError: 228334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 37 previous similar messages [ 6704.348458] Lustre: server umount lustre-OST0000 complete [ 6714.578065] Lustre: server umount lustre-MDT0000 complete [ 6718.285516] LustreError: 226479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819185 with bad export cookie 5528055222875801681 [ 6718.297890] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6718.301380] LustreError: 226479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6718.830202] Lustre: server umount lustre-MDT0001 complete [ 6740.302597] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:40:05 (1786819205) [ 6747.959677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6748.289569] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6748.338796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6748.348261] Lustre: Skipped 2 previous similar messages [ 6752.311824] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6760.361907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6760.521523] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6760.562607] Lustre: 233182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6765.631353] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6773.159353] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6778.604509] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6784.734029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6785.005815] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6785.038938] Lustre: Permanently deactivating lustre-OST0000 [ 6785.056709] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6790.428781] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6794.296669] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6797.780628] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6804.324340] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6804.369221] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6810.369766] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6812.877983] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6818.083563] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6819.518200] Lustre: Permanently reactivating lustre-OST0000 [ 6819.535084] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6819.539523] Lustre: Skipped 2 previous similar messages [ 6825.093937] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6825.107712] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 31 seconds. I think it's dead, and I am evicting it. exp 00000000f44beadf, cur 1786819292 deadline 1786819291 last 1786819261 [ 6825.172251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6833.633043] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6833.637058] LustreError: Skipped 6 previous similar messages [ 6835.534657] Lustre: server umount lustre-OST0001 complete [ 6845.486384] Lustre: server umount lustre-OST0000 complete [ 6849.062569] Lustre: server umount lustre-MDT0000 complete [ 6852.935165] LustreError: 233176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819319 with bad export cookie 5528055222875803354 [ 6852.954535] LustreError: 233176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6852.974422] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6859.454923] Lustre: server umount lustre-MDT0001 complete [ 6869.061306] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 6871.598907] Key type lgssc unregistered [ 6871.805952] LNet: 238062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6871.810331] LNetError: 238062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6871.818964] LNet: Removed LNI 192.168.203.127@tcp [ 6872.502754] Key type .llcrypt unregistered [ 6872.505250] Key type ._llcrypt unregistered [ 6893.389416] Key type ._llcrypt registered [ 6893.391931] Key type .llcrypt registered [ 6893.476390] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 6894.424233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6894.592195] alg: No test for adler32 (adler32-zlib) [ 6895.698600] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 6895.988408] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 6897.631353] Key type lgssc registered [ 6898.707617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6908.842984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6908.888766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.447704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6914.277771] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6920.673719] LustreError: 239528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6920.694780] LustreError: 239528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6922.562534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.668815] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6922.687809] Lustre: 239515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6922.701209] Lustre: Permanently deactivating lustre-MDT0001 [ 6922.712809] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6922.961806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6925.022471] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6925.046608] LustreError: 239558:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6925.062753] LustreError: 239558:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6925.081897] LustreError: 239558:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 6927.912863] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6935.427599] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6941.457066] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6948.074044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6948.409348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6954.312272] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6955.562892] LustreError: 241372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6955.568193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6960.613157] LustreError: 241372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6962.511872] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6965.737611] LustreError: 241801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6967.803779] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6970.156117] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.513157] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6975.996557] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6975.998377] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6977.624964] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6985.885783] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6989.410349] Lustre: Permanently reactivating lustre-MDT0001 [ 6989.448785] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6989.454499] Lustre: Skipped 2 previous similar messages [ 6992.501212] Lustre: lustre-MDT0001: Client d91e6e7b-f36d-4a87-943d-330cccc1ab14 (at 192.168.203.27@tcp) reconnecting [ 6997.931882] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6999.525576] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7005.331633] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7005.512983] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7009.933950] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7010.127807] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7012.631730] Lustre: Permanently deactivating lustre-MDT0001 [ 7012.654265] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7012.664585] Lustre: Skipped 2 previous similar messages [ 7020.149612] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7030.754874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7030.764989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7030.782242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7032.295844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7032.304791] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7032.321773] Lustre: Skipped 1 previous similar message [ 7035.062535] Lustre: server umount lustre-MDT0000 complete [ 7037.411346] LustreError: 239522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7037.445285] LustreError: 239522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7038.719336] LustreError: 239510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819505 with bad export cookie 2387207625184084336 [ 7038.738463] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7038.744616] LustreError: 239510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7039.029380] Lustre: server umount lustre-MDT0001 complete [ 7048.626220] Lustre: server umount lustre-OST0000 complete [ 7059.410085] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:45:25 (1786819525) [ 7067.935556] Lustre: server umount lustre-OST0001 complete [ 7073.657740] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 7082.283356] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7116.726872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7116.998197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7117.009550] Lustre: Skipped 2 previous similar messages [ 7117.037368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7117.096749] Lustre: lustre-MDT0000: new disk, initializing [ 7117.152556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7117.160180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7121.855426] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7135.020480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7135.141820] Lustre: 247833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7135.223241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7135.228688] Lustre: Skipped 1 previous similar message [ 7135.340149] Lustre: lustre-MDT0001: new disk, initializing [ 7135.461801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7135.566224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7135.591468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7140.908503] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7151.086305] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7157.345302] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7164.429066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7164.712605] Lustre: lustre-OST0000: new disk, initializing [ 7164.718129] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7164.726335] Lustre: 249801:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7164.844260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7170.326634] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7171.169375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7171.180931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7171.287430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7178.682755] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7180.429071] LustreError: 247840:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7188.297616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7188.526823] Lustre: lustre-OST0001: new disk, initializing [ 7188.532019] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7188.538646] Lustre: 250918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7190.495097] LustreError: 247840:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7190.526988] LustreError: 247839:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7194.525584] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7197.763803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7197.773625] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7197.847238] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7200.591117] LustreError: 247839:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7203.097312] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7204.379107] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7208.423750] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7208.436931] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7208.443629] Lustre: Skipped 2 previous similar messages [ 7208.444940] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7208.446687] Lustre: Skipped 3 previous similar messages [ 7212.485476] Lustre: server umount lustre-OST0001 complete [ 7213.542848] LustreError: 249786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7213.553125] LustreError: 249786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7232.993573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7232.998941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7233.003844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7233.005880] Lustre: Skipped 1 previous similar message [ 7233.011848] Lustre: Skipped 3 previous similar messages [ 7234.024610] LustreError: 249785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7234.046713] LustreError: 249785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7235.987708] Lustre: server umount lustre-OST0000 complete [ 7243.235169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7243.264715] LustreError: Skipped 1 previous similar message [ 7243.289708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7243.312839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7243.319140] Lustre: Skipped 2 previous similar messages [ 7246.485850] Lustre: server umount lustre-MDT0000 complete [ 7250.626472] LustreError: 247827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819717 with bad export cookie 2387207625184087150 [ 7250.653869] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7256.950939] Lustre: server umount lustre-MDT0001 complete [ 7269.109743] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7271.409669] Key type lgssc unregistered [ 7271.714810] LNet: 252951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7271.725189] LNetError: 252951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7271.738918] LNet: Removed LNI 192.168.203.127@tcp [ 7272.635175] Key type .llcrypt unregistered [ 7272.638081] Key type ._llcrypt unregistered [ 7307.313376] Key type ._llcrypt registered [ 7307.317819] Key type .llcrypt registered [ 7307.394352] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7308.123221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7308.132379] alg: No test for adler32 (adler32-zlib) [ 7309.127446] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 7309.348782] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 7311.007228] Key type lgssc registered [ 7312.078015] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7320.634848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7320.654094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7321.920467] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7321.938344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7322.073170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7326.023619] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7335.147240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.247841] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7335.301444] Lustre: 255140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7335.573335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7340.690530] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7348.221078] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7354.025962] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7360.332560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7360.585220] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7360.864347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7366.464918] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7367.829051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7375.405598] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7429.089189] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7429.100932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7429.125151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7429.606000] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7429.609150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7429.633769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7433.366527] Lustre: server umount lustre-OST0000 complete [ 7437.485086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7449.873936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7450.214102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7455.210128] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7455.236483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7455.254079] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7455.266950] Lustre: Skipped 1 previous similar message [ 7456.616853] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7465.332828] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7506.400271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7506.403925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7506.416263] LustreError: Skipped 1 previous similar message [ 7506.416887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7506.441729] Lustre: Skipped 1 previous similar message [ 7511.527602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7511.538291] Lustre: Skipped 1 previous similar message [ 7511.858344] Lustre: server umount lustre-OST0000 complete [ 7516.643564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7516.650745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7516.660750] Lustre: Skipped 1 previous similar message [ 7516.670300] Lustre: Skipped 1 previous similar message [ 7521.603927] Lustre: server umount lustre-MDT0000 complete [ 7521.765815] LustreError: 257738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7521.794165] LustreError: 257738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7525.031789] LustreError: 255883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819991 with bad export cookie 3737547729929979923 [ 7525.047132] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7525.049064] LustreError: 255883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7525.453548] Lustre: server umount lustre-MDT0001 complete [ 7534.194773] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7536.895410] Key type lgssc unregistered [ 7537.171475] LNet: 260564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7537.177751] LNetError: 260564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7538.221718] LNet: Removed LNI 192.168.203.127@tcp [ 7539.190991] Key type .llcrypt unregistered [ 7539.198748] Key type ._llcrypt unregistered [ 7570.147526] Key type ._llcrypt registered [ 7570.150149] Key type .llcrypt registered [ 7570.263291] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7571.695036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7571.753246] alg: No test for adler32 (adler32-zlib) [ 7573.039356] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 7573.455037] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 7575.151319] Key type lgssc registered [ 7576.156201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7583.823956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7583.838530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7585.293995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7589.084704] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7595.492294] LustreError: 261982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7595.522806] LustreError: 261982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7596.442724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7596.771443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7600.388968] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7607.008620] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7614.216823] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7621.455496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7621.610680] LustreError: 263804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7621.900417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7630.251436] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7630.338944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7638.939556] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7663.086356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7663.096257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.112504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7666.148846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7666.149543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7666.149973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7668.030782] Lustre: server umount lustre-OST0000 complete [ 7671.275864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7671.293256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7671.302759] Lustre: Skipped 2 previous similar messages [ 7673.313971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7673.326489] Lustre: Skipped 1 previous similar message [ 7677.765458] Lustre: server umount lustre-MDT0000 complete [ 7681.505944] LustreError: 265370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7681.530288] LustreError: 265370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7682.121987] LustreError: 261962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820149 with bad export cookie 11632474175569740267 [ 7682.130050] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7682.133803] LustreError: 261962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7682.491131] Lustre: server umount lustre-MDT0001 complete [ 7693.038408] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7696.823593] Key type lgssc unregistered [ 7697.080763] LNet: 266292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7697.086077] LNetError: 266292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7697.098809] LNet: Removed LNI 192.168.203.127@tcp [ 7697.835412] Key type .llcrypt unregistered [ 7697.837307] Key type ._llcrypt unregistered [ 7715.111048] Key type ._llcrypt registered [ 7715.114173] Key type .llcrypt registered [ 7715.211430] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7715.903036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7716.126143] alg: No test for adler32 (adler32-zlib) [ 7717.099778] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 7717.257440] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 7718.888571] Key type lgssc registered [ 7720.169256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7727.039756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7727.055320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7728.487872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.811787] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7738.853909] LustreError: 267517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7738.873949] LustreError: 267517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7741.154687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7741.508295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7746.590831] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7754.620806] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7760.238511] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7766.852018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7766.955969] LustreError: 269337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7767.233896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7772.357677] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7773.699428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7779.941403] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7793.120699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7793.130189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7793.153202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7794.146110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7794.146462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7794.147285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7797.103068] Lustre: server umount lustre-OST0000 complete [ 7802.848215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7802.855155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7802.867684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7802.871713] Lustre: Skipped 1 previous similar message [ 7807.920106] Lustre: server umount lustre-MDT0000 complete [ 7809.508266] LustreError: 267516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7809.545701] LustreError: 267516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7812.205098] LustreError: 267497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820279 with bad export cookie 12868252953719857423 [ 7812.220567] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7812.960547] Lustre: server umount lustre-MDT0001 complete [ 7823.372890] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7826.417827] Key type lgssc unregistered [ 7826.682266] LNet: 271199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7826.708378] LNetError: 271199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7826.735550] LNet: Removed LNI 192.168.203.127@tcp [ 7827.736824] Key type .llcrypt unregistered [ 7827.740317] Key type ._llcrypt unregistered [ 7853.982387] Key type ._llcrypt registered [ 7853.986910] Key type .llcrypt registered [ 7854.231491] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7855.885982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7855.932830] alg: No test for adler32 (adler32-zlib) [ 7857.013627] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 7857.272403] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 7859.023163] Key type lgssc registered [ 7860.499621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7869.325848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7869.382130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7870.956875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7875.245867] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7880.177619] LustreError: 272617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7880.213424] LustreError: 272617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7884.380229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7884.666043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7889.473994] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7897.839795] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7904.295358] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7910.952160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7911.335436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7915.453127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7917.414209] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7926.104611] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7955.425162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7955.435365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7955.440914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7955.466090] Lustre: Skipped 1 previous similar message [ 7955.485238] Lustre: Skipped 1 previous similar message [ 7960.438176] Lustre: server umount lustre-OST0000 complete [ 7965.683189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7965.696398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7965.696590] Lustre: Skipped 1 previous similar message [ 7970.242651] Lustre: server umount lustre-MDT0000 complete [ 7970.789196] LustreError: 272617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7970.805331] LustreError: 272617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7973.569250] LustreError: 272597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820440 with bad export cookie 15260087912671985851 [ 7973.572729] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7973.584367] LustreError: 272597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7974.081671] Lustre: server umount lustre-MDT0001 complete [ 7984.936519] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7988.229359] Key type lgssc unregistered [ 7988.601746] LNet: 276927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7988.607355] LNetError: 276927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7988.629054] LNet: Removed LNI 192.168.203.127@tcp [ 7989.480951] Key type .llcrypt unregistered [ 7989.482864] Key type ._llcrypt unregistered [ 8012.795961] Key type ._llcrypt registered [ 8012.797648] Key type .llcrypt registered [ 8013.078296] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8015.410763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8015.633320] alg: No test for adler32 (adler32-zlib) [ 8017.298596] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 8017.748427] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 8019.473821] Key type lgssc registered [ 8022.193124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8032.501322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8032.516994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8034.327840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8040.140800] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8044.515047] LustreError: 278179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8044.548876] LustreError: 278179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8049.637479] LustreError: 278181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.124022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8050.777127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8056.727454] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8065.044644] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8071.195666] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8078.207103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8078.581360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8079.618820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8085.380113] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8093.225656] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8110.560904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8110.575143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8110.591964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8111.587981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8111.590328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8111.593408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8111.618510] Lustre: Skipped 1 previous similar message [ 8114.260500] Lustre: server umount lustre-OST0000 complete [ 8121.825720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8121.830969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8121.845939] Lustre: Skipped 1 previous similar message [ 8124.366705] Lustre: server umount lustre-MDT0000 complete [ 8126.960744] LustreError: 278182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8126.974301] LustreError: 278182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8127.696467] LustreError: 278144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820594 with bad export cookie 8950724333690859834 [ 8127.703426] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8127.711749] LustreError: 278144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8128.053571] Lustre: server umount lustre-MDT0001 complete [ 8139.449320] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 8143.248932] Key type lgssc unregistered [ 8143.583864] LNet: 281886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8143.592788] LNetError: 281886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8143.616817] LNet: Removed LNI 192.168.203.127@tcp [ 8144.670274] Key type .llcrypt unregistered [ 8144.674072] Key type ._llcrypt unregistered [ 8156.860847] Key type ._llcrypt registered [ 8156.865647] Key type .llcrypt registered [ 8156.962396] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8174.314436] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 8189.060877] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8190.427288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8190.473061] alg: No test for adler32 (adler32-zlib) [ 8191.492695] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 8191.704662] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 8193.423141] Key type lgssc registered [ 8194.747369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8233.344866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8233.366598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8234.748902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8234.838386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8235.041927] Lustre: lustre-MDT0000: new disk, initializing [ 8235.154701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8235.165771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8240.952851] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8252.037830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8252.196691] Lustre: 285437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8252.246868] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8252.251154] Lustre: Skipped 1 previous similar message [ 8252.308818] Lustre: lustre-MDT0001: new disk, initializing [ 8252.397180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8252.450490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8252.465085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8256.755192] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8266.560991] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8272.999521] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8279.428470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8279.747302] Lustre: lustre-OST0000: new disk, initializing [ 8279.751521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8279.759412] Lustre: 287407:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8279.878948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8285.418925] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8289.876556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8289.919724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8290.143987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8295.846547] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8301.342607] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8301.578953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8306.965667] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8307.123876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8310.759463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8310.775889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8310.780383] Lustre: Skipped 1 previous similar message [ 8310.796867] Lustre: Skipped 1 previous similar message [ 8315.312073] Lustre: server umount lustre-OST0000 complete [ 8320.993334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8321.002981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8321.003807] Lustre: Skipped 1 previous similar message [ 8321.020166] Lustre: Skipped 1 previous similar message [ 8325.997714] Lustre: server umount lustre-MDT0000 complete [ 8330.494890] LustreError: 287406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820797 with bad export cookie 16854762512243577413 [ 8330.497193] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8330.506871] LustreError: 287406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8330.906606] Lustre: server umount lustre-MDT0001 complete [ 8340.683703] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:06:46 (1786820806) [ 8346.770634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.341788] LustreError: 289818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8347.376271] LustreError: 289818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8347.460765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8352.484352] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8352.740193] LustreError: 289819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8357.857260] LustreError: 289818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8360.789444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8361.118720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8365.293647] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8373.659841] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8379.994470] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8386.589689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8386.941796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8394.050769] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8402.032206] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8415.712488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8415.714962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8415.715429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8415.722216] Lustre: Skipped 1 previous similar message [ 8420.426877] Lustre: server umount lustre-OST0000 complete [ 8425.952669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8425.966336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8425.971929] Lustre: Skipped 2 previous similar messages [ 8431.074037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8431.081963] Lustre: Skipped 1 previous similar message [ 8438.239171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8438.475821] Lustre: server umount lustre-MDT0000 complete [ 8440.297436] LustreError: 290548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8440.333100] LustreError: 290548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8442.674491] LustreError: 290556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820909 with bad export cookie 16854762512243578288 [ 8442.676892] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8442.688461] LustreError: 290556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8443.061267] Lustre: server umount lustre-MDT0001 complete [ 8453.351505] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 8456.413339] Key type lgssc unregistered [ 8456.739854] LNet: 293361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8456.753596] LNetError: 293361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8457.765854] LNet: Removed LNI 192.168.203.127@tcp [ 8458.840410] Key type .llcrypt unregistered [ 8458.842889] Key type ._llcrypt unregistered [ 8491.799689] Key type ._llcrypt registered [ 8491.802248] Key type .llcrypt registered [ 8491.883556] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 8506.521925] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8507.504527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8507.535276] alg: No test for adler32 (adler32-zlib) [ 8508.696817] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 8508.980034] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 8510.687195] Key type lgssc registered [ 8511.939564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8556.520062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8556.559761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8557.805122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8557.830934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8557.886450] Lustre: lustre-MDT0000: new disk, initializing [ 8557.947537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8557.966963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8561.976354] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8572.623778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8572.787324] Lustre: 297190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8572.836041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8572.839150] Lustre: Skipped 1 previous similar message [ 8572.913563] Lustre: lustre-MDT0001: new disk, initializing [ 8573.036933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8573.072886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8573.097453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8577.594077] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8588.042788] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8593.735991] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8600.763602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8601.000078] Lustre: lustre-OST2710: new disk, initializing [ 8601.004524] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8601.025909] Lustre: 299156:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8601.142022] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8606.310068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8606.334448] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8606.449158] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8607.153903] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8616.512849] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8626.312389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8626.559625] Lustre: lustre-OST03e8: new disk, initializing [ 8626.569271] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8626.589295] Lustre: 300221:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8626.785373] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8633.733113] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8634.516354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8634.525915] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8634.666944] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8641.932734] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8650.603737] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8651.024924] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8656.419290] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8656.855845] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8663.129847] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8663.369310] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8668.536395] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8668.859333] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8675.298258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8675.308159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8675.809157] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8675.820985] Lustre: Skipped 1 previous similar message [ 8676.524977] Lustre: server umount lustre-MDT0000 complete [ 8680.228436] LustreError: 297181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821147 with bad export cookie 5952752678967989604 [ 8680.240240] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8680.241084] LustreError: 297181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8680.543181] Lustre: server umount lustre-MDT0001 complete [ 8695.555670] Lustre: server umount lustre-OST2710 complete [ 8696.288200] Lustre: 294987:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786821147/real 1786821147] req@00000000f825cc07 x1873617590751744/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786821163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8696.336611] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8696.350371] Lustre: Skipped 1 previous similar message [ 8699.922644] Lustre: server umount lustre-OST03e8 complete [ 8716.113686] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 8725.417408] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8758.635085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8758.889393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8758.915926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8759.013440] Lustre: lustre-MDT0000: new disk, initializing [ 8759.171066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8759.192454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8763.974411] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8774.146682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8774.294742] Lustre: 305471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8774.350333] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8774.354504] Lustre: Skipped 1 previous similar message [ 8774.431586] Lustre: lustre-MDT0001: new disk, initializing [ 8774.533302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8774.576361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8774.600249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8779.318725] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8788.545546] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8793.838746] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8800.096265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8800.364765] Lustre: lustre-OST0000: new disk, initializing [ 8800.373847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8800.402714] Lustre: 307444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8800.470760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8805.974214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8805.992168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8806.071714] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8806.082633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8814.270917] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8817.505921] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8817.674753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8821.248417] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8821.463487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8826.849407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8826.849956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8826.866920] LustreError: Skipped 1 previous similar message [ 8826.888525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8829.471934] Lustre: server umount lustre-OST0000 complete [ 8836.064407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8836.071676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8836.086244] Lustre: Skipped 1 previous similar message [ 8836.093543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8836.097806] Lustre: Skipped 1 previous similar message [ 8842.211099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8842.217549] Lustre: Skipped 3 previous similar messages [ 8846.310206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8846.317034] Lustre: Skipped 1 previous similar message [ 8847.327459] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8847.616697] Lustre: server umount lustre-MDT0000 complete [ 8850.772965] LustreError: 305465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821317 with bad export cookie 5952752678967991298 [ 8850.781943] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8850.784021] LustreError: 305465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8851.051429] Lustre: server umount lustre-MDT0001 complete [ 8858.440612] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:15:24 (1786821324) [ 8859.559914] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8861.178219] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:15:26 (1786821326) [ 8864.435485] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8887.348678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8887.566130] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8887.588859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8887.709909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8891.362934] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8898.186265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8898.249703] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8898.271413] Lustre: 311295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8902.212259] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8922.150170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8922.341942] Lustre: Denying initial registration attempt from nid 192.168.203.127@tcp, specified as failover [ 8922.358713] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8922.379831] LustreError: 312750:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8922.392858] LustreError: 312750:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8922.401423] LustreError: 312750:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8922.481462] Lustre: server umount lustre-OST0000 complete [ 8922.485918] LustreError: 312750:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8934.375697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8934.390447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8934.413523] Lustre: Skipped 1 previous similar message [ 8934.423178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8936.156170] Lustre: server umount lustre-MDT0000 complete [ 8939.497808] LustreError: 311289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821406 with bad export cookie 5952752678967992180 [ 8939.503266] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8939.508377] LustreError: 311289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8939.786460] Lustre: server umount lustre-MDT0001 complete [ 8950.601613] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 8952.959260] Key type lgssc unregistered [ 8953.199482] LNet: 313874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8953.211716] LNetError: 313874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8953.230178] LNet: Removed LNI 192.168.203.127@tcp [ 8954.178926] Key type .llcrypt unregistered [ 8954.182292] Key type ._llcrypt unregistered [ 8965.121507] Key type ._llcrypt registered [ 8965.123163] Key type .llcrypt registered [ 8965.226281] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8966.041753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8966.069146] alg: No test for adler32 (adler32-zlib) [ 8967.053131] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 8967.303398] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 8968.983154] Key type lgssc registered [ 8970.246571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9014.221767] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 9022.852683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9022.880176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9024.108167] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9024.128689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9024.312146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9028.693818] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9038.222113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.369507] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9038.422214] Lustre: 316621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9038.960977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9043.990741] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9055.077811] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9060.708560] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9080.369148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9080.558244] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9080.756576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9086.508179] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9094.219459] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9097.698611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9097.700401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9097.708671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9097.718089] Lustre: Skipped 1 previous similar message [ 9097.728264] Lustre: Skipped 1 previous similar message [ 9102.266199] Lustre: server umount lustre-OST0000 complete [ 9107.938415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9107.946076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9107.965569] Lustre: Skipped 1 previous similar message [ 9107.986933] Lustre: Skipped 1 previous similar message [ 9113.056921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9113.072820] Lustre: Skipped 1 previous similar message [ 9117.155344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9121.247659] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9121.492848] Lustre: server umount lustre-MDT0000 complete [ 9125.103133] LustreError: 316613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821592 with bad export cookie 18400930975385434581 [ 9125.106230] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9125.118401] LustreError: 316613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9125.439208] Lustre: server umount lustre-MDT0001 complete [ 9135.642489] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 9139.366073] Key type lgssc unregistered [ 9139.769760] LNet: 320199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9139.784544] LNetError: 320199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9139.817846] LNet: Removed LNI 192.168.203.127@tcp [ 9141.005330] Key type .llcrypt unregistered [ 9141.010551] Key type ._llcrypt unregistered [ 9166.175282] Key type ._llcrypt registered [ 9166.177727] Key type .llcrypt registered [ 9166.290425] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 9168.097850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9168.139083] alg: No test for adler32 (adler32-zlib) [ 9169.259195] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 9169.554468] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 9171.279384] Key type lgssc registered [ 9172.780683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9181.247377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9181.272594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9182.897323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9186.965911] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9192.418257] LustreError: 321623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9192.430503] LustreError: 321623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9196.060677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9196.486815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9200.566682] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9207.567751] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9212.702526] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9220.981980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9221.302817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9226.366665] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9232.839748] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9242.592199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9242.598871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9242.617053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9243.112768] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9243.121715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9243.133233] Lustre: Skipped 1 previous similar message [ 9243.140147] Lustre: Skipped 1 previous similar message [ 9245.989132] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 9245.991911] Lustre: Skipped 1 previous similar message [ 9248.052194] Lustre: server umount lustre-MDT0000 complete [ 9248.237878] LustreError: 324116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9248.262095] LustreError: 324116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9251.108494] LustreError: 323877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9251.914081] LustreError: 321605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821718 with bad export cookie 18406338511794636784 [ 9251.937246] LustreError: 321605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9251.938078] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9253.344961] LustreError: 321618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9253.354389] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9253.389497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9258.373906] Lustre: server umount lustre-MDT0001 complete [ 9264.246648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9272.602445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9276.895795] LustreError: 320932:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005108a4dd x1873618283828352/t0(0) o250->MGC192.168.203.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9277.206085] LustreError: 323453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9277.236708] LustreError: 323453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9277.308759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9277.370552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9282.694930] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9286.497663] LustreError: 325215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9286.500241] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9286.545143] LustreError: 325215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9292.494482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9292.728956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9296.904392] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9297.917163] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9305.706633] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9307.343227] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9312.934724] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9315.024717] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9323.487922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9323.493681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9323.497919] LustreError: Skipped 1 previous similar message [ 9323.499478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9323.499483] Lustre: Skipped 2 previous similar messages [ 9323.555379] Lustre: Skipped 1 previous similar message [ 9325.968403] Lustre: server umount lustre-OST0000 complete [ 9333.730820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9333.736331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9333.745698] Lustre: Skipped 1 previous similar message [ 9333.759952] Lustre: Skipped 1 previous similar message [ 9337.109267] Lustre: server umount lustre-MDT0000 complete [ 9338.853302] LustreError: 325216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9338.917195] LustreError: 325216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9341.555101] LustreError: 321605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821808 with bad export cookie 18406338511794644414 [ 9341.557846] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9341.562607] LustreError: 321605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9341.854525] Lustre: server umount lustre-MDT0001 complete [ 9353.815514] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 9356.814994] Key type lgssc unregistered [ 9357.196740] LNet: 327894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9357.202029] LNetError: 327894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9358.246547] LNet: Removed LNI 192.168.203.127@tcp [ 9358.992523] Key type .llcrypt unregistered [ 9358.994332] Key type ._llcrypt unregistered [ 9382.374343] Key type ._llcrypt registered [ 9382.378275] Key type .llcrypt registered [ 9382.533545] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 9383.664029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9383.735690] alg: No test for adler32 (adler32-zlib) [ 9384.845635] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 9385.097167] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 9386.759701] Key type lgssc registered [ 9387.820104] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9394.935179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9394.951243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9396.260292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9400.279258] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9406.438363] LustreError: 329313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9406.471776] LustreError: 329313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9408.195634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9408.525490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9412.615853] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9419.478640] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9424.518441] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9428.961570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9428.970281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9428.981399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9432.033138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9432.051096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9432.066564] Lustre: Skipped 1 previous similar message [ 9434.088411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9434.094033] Lustre: Skipped 1 previous similar message [ 9439.201343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9439.208197] Lustre: Skipped 1 previous similar message [ 9442.273101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9442.370062] Lustre: server umount lustre-MDT0000 complete [ 9444.334571] LustreError: 330352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9444.365820] LustreError: 330352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9445.966580] LustreError: 330046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821912 with bad export cookie 17050075766412211841 [ 9445.978764] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9445.979654] LustreError: 330046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9446.243223] Lustre: server umount lustre-MDT0001 complete [ 9461.120475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9461.489257] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9461.566879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9461.797658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9466.841212] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9475.017510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9475.102156] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9475.134520] Lustre: 331903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9475.369825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9479.459986] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9487.047270] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9493.881848] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9504.493043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9504.724373] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9504.736061] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9504.748461] LustreError: 333788:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9504.754337] LustreError: 333788:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9504.759412] LustreError: 333788:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9504.840596] Lustre: server umount lustre-OST0000 complete [ 9504.843415] LustreError: 333788:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9515.117850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9515.297710] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9515.504857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9520.939916] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9521.843330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9529.676628] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9537.959064] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9538.094149] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9538.173785] Lustre: lustre-OST0001: new disk, initializing [ 9538.178327] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9538.192944] Lustre: 335328:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9538.310340] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9544.809450] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9546.916516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9546.931128] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9547.069596] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9553.559884] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9557.472094] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9557.472475] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9557.478711] LustreError: Skipped 1 previous similar message [ 9557.493844] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9557.497270] Lustre: Skipped 1 previous similar message [ 9562.800985] Lustre: server umount lustre-OST0001 complete [ 9567.720789] LustreError: 334660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9567.721047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9567.728131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9567.728139] Lustre: Skipped 2 previous similar messages [ 9567.747627] LustreError: 334660:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9567.794946] Lustre: Skipped 2 previous similar messages [ 9572.462879] Lustre: server umount lustre-OST0000 complete [ 9577.955470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9577.956082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9577.975284] Lustre: Skipped 1 previous similar message [ 9587.170115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9587.177654] Lustre: Skipped 6 previous similar messages [ 9590.239129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9590.464808] Lustre: server umount lustre-MDT0000 complete [ 9592.288811] LustreError: 332848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9592.303446] LustreError: 332848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9593.982490] LustreError: 331893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822060 with bad export cookie 17050075766412212394 [ 9593.985323] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9594.000435] LustreError: 331893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9594.327857] Lustre: server umount lustre-MDT0001 complete [ 9603.829176] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 9606.482606] Key type lgssc unregistered [ 9606.744787] LNet: 337290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9606.752278] LNetError: 337290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9606.769304] LNet: Removed LNI 192.168.203.127@tcp [ 9607.449677] Key type .llcrypt unregistered [ 9607.454786] Key type ._llcrypt unregistered [ 9672.410524] Key type ._llcrypt registered [ 9672.411992] Key type .llcrypt registered [ 9672.522667] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 9684.940426] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 9686.165404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9686.188034] alg: No test for adler32 (adler32-zlib) [ 9687.218712] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [ 9687.443505] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 9689.184741] Key type lgssc registered [ 9690.189463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9724.022693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9724.063062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9725.266293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9725.289272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9725.362751] Lustre: lustre-MDT0000: new disk, initializing [ 9725.422366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9725.453956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9729.700950] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9739.706843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9739.854505] Lustre: 342204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9739.927204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9739.932545] Lustre: Skipped 1 previous similar message [ 9740.039974] Lustre: lustre-MDT0001: new disk, initializing [ 9740.154642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9740.187430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9740.195542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9744.363872] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9754.858517] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9761.385988] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9768.987193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9769.505091] Lustre: lustre-OST0000: new disk, initializing [ 9769.512228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9769.520407] Lustre: 344174:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9769.638238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9776.170521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9776.178947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9776.351052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9776.588445] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9785.478740] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9789.819454] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9790.060465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9795.226515] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9795.397626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9801.698577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9801.717676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9801.741357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9802.209740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9802.228366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9803.613659] Lustre: server umount lustre-OST0000 complete [ 9807.329183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9807.339131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9807.347950] Lustre: Skipped 1 previous similar message [ 9807.366753] Lustre: Skipped 3 previous similar messages [ 9812.448928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9812.452199] Lustre: Skipped 1 previous similar message [ 9813.446304] Lustre: server umount lustre-MDT0000 complete [ 9816.929604] LustreError: 342197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822283 with bad export cookie 5238851948603812022 [ 9816.931377] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9816.935747] LustreError: 342197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9817.410738] Lustre: server umount lustre-MDT0001 complete [ 9828.511090] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:31:33 (1786822293) [ 9841.024254] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:31:46 (1786822306) [ 9850.033720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9850.567374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9854.890234] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9855.969482] LustreError: 346929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9855.987777] LustreError: 346929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9861.089088] LustreError: 346930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9863.898289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9864.298208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9868.522024] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9875.829193] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9881.937689] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9888.961958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9889.264692] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 9889.385762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9895.293064] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9902.170243] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9909.174656] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9911.293293] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9912.868871] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9914.567436] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9916.354055] Lustre: Failing over lustre-MDT0000 [ 9916.602866] Lustre: server umount lustre-MDT0000 complete [ 9916.913188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9916.920261] LustreError: 349421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9916.930675] Lustre: Skipped 2 previous similar messages [ 9916.966056] LustreError: 349421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9919.286766] LustreError: 349421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9919.309671] LustreError: 349421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9924.364324] LustreError: 348658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.392195] LustreError: 348658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9934.618947] LustreError: 348658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9934.662273] LustreError: 348658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9935.788953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9935.888688] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9936.128738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9936.188163] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9940.313354] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9940.753066] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9941.506207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9941.595877] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9941.647733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9949.990863] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9951.470210] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9954.832717] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9956.656568] Lustre: Failing over lustre-MDT0000 [ 9956.811714] Lustre: server umount lustre-MDT0000 complete [ 9956.843300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9956.863147] LustreError: 348658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9956.883114] LustreError: 348658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9974.237092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9974.350162] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9974.530351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9974.570581] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9976.594845] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9978.297790] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9979.884764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9979.895904] Lustre: Skipped 2 previous similar messages [ 9979.925436] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9979.986345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9986.999912] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9988.535727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9995.233845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9995.236167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9995.242342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9995.261047] Lustre: Skipped 4 previous similar messages [ 9999.684228] Lustre: server umount lustre-MDT0000 complete [10000.354737] LustreError: 346929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10000.401907] LustreError: 346929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [10003.662304] LustreError: 346911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822470 with bad export cookie 5238851948603814731 [10003.669991] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10003.673602] LustreError: 346911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10004.018333] Lustre: server umount lustre-MDT0001 complete [10018.538016] Lustre: server umount lustre-OST0000 complete [10037.665209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10039.019786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10046.476778] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10055.530412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.452335] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10067.729465] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10074.188044] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10083.992210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10091.441122] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10098.992445] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10109.102332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10114.534417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10114.542441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10114.559183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10114.566512] Lustre: Skipped 1 previous similar message [10114.585930] Lustre: Skipped 3 previous similar messages [10116.644294] Lustre: server umount lustre-OST0000 complete [10122.722575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10126.738238] Lustre: server umount lustre-MDT0000 complete [10129.891382] LustreError: 353995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.906165] LustreError: 353995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10130.837384] LustreError: 355112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822597 with bad export cookie 5238851948603815480 [10130.845522] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10130.852209] LustreError: 355112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10131.136151] Lustre: server umount lustre-MDT0001 complete [10143.140365] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [10145.697600] Key type lgssc unregistered [10145.923058] LNet: 356825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10145.940853] LNetError: 356825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10145.965255] LNet: Removed LNI 192.168.203.127@tcp [10147.024609] Key type .llcrypt unregistered [10147.026453] Key type ._llcrypt unregistered [10175.297608] Key type ._llcrypt registered [10175.300407] Key type .llcrypt registered [10175.414205] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [10176.782365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10176.833513] alg: No test for adler32 (adler32-zlib) [10177.933436] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [10178.254810] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [10179.920406] Key type lgssc registered [10181.481378] Lustre: Echo OBD driver; http://www.lustre.org/ [10191.990439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10192.037434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10193.496075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10198.267642] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10203.632611] LustreError: 358343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10203.661901] LustreError: 358343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10208.736562] LustreError: 358342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10208.770484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10209.161645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10213.826730] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10222.875206] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10230.180799] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10238.411657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10238.636315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10239.680701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10244.088572] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10251.915841] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10264.547283] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10264.549730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10264.578613] Lustre: Skipped 1 previous similar message [10266.498832] Lustre: server umount lustre-MDT0000 complete [10269.668180] LustreError: 358342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10269.683197] LustreError: 358342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10270.392837] LustreError: 358324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822737 with bad export cookie 5841653308565393743 [10270.398615] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10270.410366] LustreError: 358324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10270.857782] Lustre: server umount lustre-MDT0001 complete [10284.990840] Lustre: server umount lustre-OST0000 complete [10298.030595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10298.392791] LustreError: 362052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10298.480727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10302.628375] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10303.975170] LustreError: 362053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10312.618147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10313.149429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10316.889351] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10324.200948] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10329.394522] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10335.934861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10336.236905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10342.928886] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10344.433871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10349.901794] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10364.384959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10364.395570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10364.415590] Lustre: Skipped 1 previous similar message [10364.424182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10364.426776] Lustre: Skipped 2 previous similar messages [10366.804457] Lustre: server umount lustre-MDT0000 complete [10370.004460] LustreError: 362034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822836 with bad export cookie 5841653308565394856 [10370.020623] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10370.022301] LustreError: 362034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10370.042920] LustreError: 364194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10370.043293] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10370.061263] LustreError: 364194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10370.081211] Lustre: Skipped 2 previous similar messages [10370.084656] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10370.088208] Lustre: Skipped 2 previous similar messages [10370.471532] Lustre: server umount lustre-MDT0001 complete [10376.864060] Lustre: server umount lustre-OST0000 complete [10393.169083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10393.859118] LustreError: 365962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10394.015146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10399.603854] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10410.254897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10414.567189] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10422.473960] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10427.699324] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10434.103376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10434.392381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10434.397516] Lustre: Skipped 1 previous similar message [10438.513965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10439.785273] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10446.869690] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10453.245096] Lustre: server umount lustre-OST0000 complete [10459.109083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10459.110521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10459.129422] Lustre: Skipped 1 previous similar message [10459.145548] Lustre: Skipped 1 previous similar message [10464.225109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10464.233552] Lustre: Skipped 1 previous similar message [10466.359166] Lustre: server umount lustre-MDT0000 complete [10469.348232] LustreError: 366692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10469.368222] LustreError: 366692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10469.841183] Lustre: 357656:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786822920/real 1786822920] req@00000000364dab32 x1873619341892096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786822936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10469.885255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10470.572422] LustreError: 365942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822937 with bad export cookie 5841653308565395878 [10470.575406] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10470.593163] LustreError: 365942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10471.134643] Lustre: server umount lustre-MDT0001 complete [10480.513795] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [10482.932653] Key type lgssc unregistered [10483.259150] LNet: 369497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10483.267853] LNetError: 369497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10483.282867] LNet: Removed LNI 192.168.203.127@tcp [10484.063112] Key type .llcrypt unregistered [10484.064559] Key type ._llcrypt unregistered [10503.914194] Key type ._llcrypt registered [10503.921492] Key type .llcrypt registered [10504.019709] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [10504.989700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10505.029076] alg: No test for adler32 (adler32-zlib) [10506.256625] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [10506.511581] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [10508.215087] Key type lgssc registered [10509.158830] Lustre: Echo OBD driver; http://www.lustre.org/ [10516.659817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10516.685610] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10516.694995] LustreError: 370919:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10516.729178] LustreError: 370919:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10516.739544] LustreError: 370919:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10516.751143] LustreError: 370919:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10516.761806] LustreError: 370919:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10523.758886] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10523.767677] LustreError: 371316:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10523.790296] LustreError: 371316:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10523.794916] LustreError: 371316:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10523.805560] LustreError: 371316:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10523.816451] LustreError: 371316:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10541.381890] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [10544.031469] Key type lgssc unregistered [10544.297783] LNet: 372321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10544.307084] LNetError: 372321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10544.322114] LNet: Removed LNI 192.168.203.127@tcp [10545.014809] Key type .llcrypt unregistered [10545.018057] Key type ._llcrypt unregistered [10561.365425] Key type ._llcrypt registered [10561.367504] Key type .llcrypt registered [10561.500548] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [10574.150288] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [10575.122742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10575.244301] alg: No test for adler32 (adler32-zlib) [10576.380081] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [10576.621565] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [10578.271189] Key type lgssc registered [10579.401744] Lustre: Echo OBD driver; http://www.lustre.org/ [10612.112287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10612.154514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10613.375620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10613.400621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10613.466888] Lustre: lustre-MDT0000: new disk, initializing [10613.522060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10613.542967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10617.658954] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10627.613293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10627.706887] Lustre: 375455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10627.769566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10627.776646] Lustre: Skipped 1 previous similar message [10627.865438] Lustre: lustre-MDT0001: new disk, initializing [10627.960459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10627.992753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10628.004993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10631.335846] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10640.015295] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10645.428268] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10651.938157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10652.543831] Lustre: lustre-OST0000: new disk, initializing [10652.552865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10652.582916] Lustre: 377421:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10652.782588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10658.914712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10658.931518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10659.306136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10661.946171] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10674.789319] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10682.024519] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10682.346491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10687.897927] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10688.181853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10690.016796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10690.023887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10690.037614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10690.049559] Lustre: Skipped 1 previous similar message [10695.142909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10695.156883] Lustre: Skipped 2 previous similar messages [10700.270502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10700.278328] Lustre: Skipped 1 previous similar message [10704.357423] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10704.490256] Lustre: server umount lustre-OST0000 complete [10709.985077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10709.994955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10710.005204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10715.617722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10715.631258] Lustre: Skipped 3 previous similar messages [10722.273284] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10722.751917] Lustre: server umount lustre-MDT0000 complete [10727.380425] LustreError: 375444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823194 with bad export cookie 17680360987464472224 [10727.384610] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10727.391857] LustreError: 375444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10727.819394] Lustre: server umount lustre-MDT0001 complete [10738.360971] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10741.366543] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:46:45 (1786823205) [10748.826390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10749.307127] LustreError: 379886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10749.339334] LustreError: 379886:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10749.409608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10754.532878] LustreError: 379887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10754.612993] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10759.649930] LustreError: 379886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10763.439539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10763.696191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10768.505433] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10776.775317] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10783.207832] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10791.407337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10791.730225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10798.294241] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10805.380174] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10811.699327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10811.911641] Lustre: lustre-OST0001: new disk, initializing [10811.917250] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10811.924255] Lustre: 382723:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10812.069771] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10817.698703] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10818.156876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10818.177191] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10818.299099] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10825.754726] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10832.690033] Lustre: lustre-OST0001: Not available for connect from 192.168.203.27@tcp (stopping) [10832.700911] Lustre: Skipped 2 previous similar messages [10833.893485] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10833.897766] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10833.935298] Lustre: Skipped 2 previous similar messages [10835.378194] Lustre: server umount lustre-OST0001 complete [10837.776047] LustreError: 381710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10844.136726] LustreError: 381709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10844.158378] LustreError: 381709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10854.370180] LustreError: 383050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10854.377263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10854.382477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10854.392260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10854.392268] Lustre: Skipped 2 previous similar messages [10854.410156] LustreError: 383050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10860.201722] Lustre: server umount lustre-OST0000 complete [10866.149034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10866.170815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10866.197723] Lustre: Skipped 1 previous similar message [10871.244882] Lustre: server umount lustre-MDT0000 complete [10874.849085] LustreError: 380614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.870025] LustreError: 380614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10875.129479] LustreError: 379867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823342 with bad export cookie 17680360987464473106 [10875.133079] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10875.148730] LustreError: 379867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10881.585701] Lustre: server umount lustre-MDT0001 complete [10891.302785] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [10894.402033] Key type lgssc unregistered [10894.696687] LNet: 384713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10894.702445] LNetError: 384713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10894.715679] LNet: Removed LNI 192.168.203.127@tcp [10895.493414] Key type .llcrypt unregistered [10895.495051] Key type ._llcrypt unregistered [10953.406413] Key type ._llcrypt registered [10953.409606] Key type .llcrypt registered [10953.521166] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [10954.705643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10954.787073] alg: No test for adler32 (adler32-zlib) [10956.064365] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [10956.363836] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [10958.032927] Key type lgssc registered [10959.375766] Lustre: Echo OBD driver; http://www.lustre.org/ [10966.972907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10966.994879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10968.294910] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10968.324838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10968.485034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10972.465143] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10980.177399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10980.315369] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10980.356430] Lustre: 387511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10980.642863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10985.469149] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10992.195951] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10998.386352] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11001.314319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11001.332829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11001.361196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11002.851957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11002.854385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11007.969139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11007.978567] Lustre: Skipped 2 previous similar messages [11013.088794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11013.106074] Lustre: Skipped 1 previous similar message [11015.135153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11015.330806] Lustre: server umount lustre-MDT0000 complete [11018.920563] LustreError: 387503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823485 with bad export cookie 3286792471792977907 [11018.931633] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11018.939160] LustreError: 387503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11019.352841] Lustre: server umount lustre-MDT0001 complete [11030.840575] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:51:36 (1786823496) [11040.532400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11041.056449] LustreError: 390026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11041.080200] LustreError: 390026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11041.126985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11046.369420] LustreError: 390027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11046.894354] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11051.491524] LustreError: 390026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11055.038805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11055.371180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11059.763801] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11065.908401] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11071.238601] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11077.533812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11077.758939] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11077.970465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11083.391596] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11090.934881] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11098.883406] Lustre: Permanently deactivating lustre-OST0000 [11098.895194] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11104.339420] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11106.975089] LustreError: 392965:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11106.982987] LustreError: 392965:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11106.993043] LustreError: 392965:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11109.879711] LustreError: 393015:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11109.886651] LustreError: 393015:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11117.968095] Lustre: server umount lustre-OST0000 complete [11120.394648] LustreError: 393264:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11120.407175] LustreError: 393264:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11123.170463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11123.174899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11123.191480] Lustre: Skipped 1 previous similar message [11132.549984] Lustre: server umount lustre-MDT0000 complete [11133.410639] LustreError: 390023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11133.432244] LustreError: 390023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11136.375880] LustreError: 390006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823603 with bad export cookie 3286792471792978488 [11136.383911] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11136.384947] LustreError: 390006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11146.869082] Lustre: server umount lustre-MDT0001 complete [11156.707963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11163.025405] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11165.911623] LustreError: 394424:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11172.719429] Lustre: 394763:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.127@tcp [11174.762933] Lustre: server umount MGS complete [11186.276076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11186.613431] LustreError: 395487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11186.637232] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11186.650346] Lustre: Skipped 1 previous similar message [11186.705243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11191.257554] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11196.899198] LustreError: 395487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11196.913880] LustreError: 395487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11199.691623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11199.979858] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11203.872345] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11210.616790] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11215.845825] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11217.450971] Lustre: Permanently reactivating lustre-OST0000 [11217.464228] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11217.471348] Lustre: Skipped 2 previous similar messages [11247.583178] LustreError: 397364:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11255.998846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11256.344532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11256.349743] Lustre: Skipped 1 previous similar message [11262.372310] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11269.136602] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11275.865041] Lustre: server umount lustre-OST0000 complete [11281.888993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11281.899271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11281.921178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11281.928130] Lustre: Skipped 3 previous similar messages [11290.237253] Lustre: server umount lustre-MDT0000 complete [11293.151475] Lustre: 386834:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786823744/real 1786823744] req@00000000e9c000e5 x1873620157691776/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786823760 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11293.160943] LustreError: 395488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11293.180823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11293.203869] LustreError: 395488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11293.222427] Lustre: Skipped 1 previous similar message [11293.895918] LustreError: 395467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823760 with bad export cookie 3286792471792979538 [11293.902144] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11293.904211] LustreError: 395467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11294.378880] Lustre: server umount lustre-MDT0001 complete [11304.149070] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [11306.735946] Key type lgssc unregistered [11307.012715] LNet: 399996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11307.017494] LNetError: 399996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11307.032822] LNet: Removed LNI 192.168.203.127@tcp [11308.083050] Key type .llcrypt unregistered [11308.086230] Key type ._llcrypt unregistered [11327.777873] Key type ._llcrypt registered [11327.780888] Key type .llcrypt registered [11327.886639] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [11342.667901] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [11343.715842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11343.741343] alg: No test for adler32 (adler32-zlib) [11344.664934] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [11344.870386] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [11346.567240] Key type lgssc registered [11347.457774] Lustre: Echo OBD driver; http://www.lustre.org/ [11382.027774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11382.071041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11383.364927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11383.397871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11383.487268] Lustre: lustre-MDT0000: new disk, initializing [11383.607347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11383.667546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11389.807959] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11399.866189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11399.990315] Lustre: 403155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11400.045572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11400.048686] Lustre: Skipped 1 previous similar message [11400.095455] Lustre: lustre-MDT0001: new disk, initializing [11400.190411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11400.223589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11400.236790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11403.728363] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11412.643844] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11418.226647] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11424.628896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11424.926375] Lustre: lustre-OST0000: new disk, initializing [11424.930233] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11424.935846] Lustre: 405122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11425.013517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11430.987644] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11435.080389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11435.090292] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11435.196555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11439.727130] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11444.099483] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11444.295140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11448.255262] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11448.479190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11450.857298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11450.868101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11450.873029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11450.875026] Lustre: Skipped 1 previous similar message [11455.969129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11455.980446] Lustre: Skipped 2 previous similar messages [11456.380654] Lustre: server umount lustre-OST0000 complete [11461.097742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11461.121488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11461.134775] Lustre: Skipped 1 previous similar message [11466.212209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11466.217893] Lustre: Skipped 1 previous similar message [11471.338619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11471.348569] Lustre: Skipped 1 previous similar message [11474.399108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11474.643499] Lustre: server umount lustre-MDT0000 complete [11478.264721] LustreError: 403148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823945 with bad export cookie 14594289367152374279 [11478.269488] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11478.276664] LustreError: 403148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11478.760466] Lustre: server umount lustre-MDT0001 complete [11488.129411] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11490.150175] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:59:15 (1786823955) [11496.464655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11496.775000] LustreError: 407584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11496.796105] LustreError: 407584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11496.845914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11501.838186] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11502.050447] LustreError: 407585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11506.156457] LustreError: 407584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11509.550969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11509.865219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11514.016644] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11521.327314] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11526.917506] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11533.560190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11533.802020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11539.712171] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11546.528573] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11548.764370] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11551.201479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11555.308359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11555.308687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11555.318540] Lustre: Skipped 1 previous similar message [11555.332209] LustreError: Skipped 1 previous similar message [11555.338958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11561.115075] Lustre: server umount lustre-OST0000 complete [11565.542152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11565.550452] Lustre: Skipped 1 previous similar message [11570.670560] LustreError: 408312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11570.928974] Lustre: server umount lustre-MDT0000 complete [11575.777183] LustreError: 407584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.806628] LustreError: 407584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11576.236473] LustreError: 408319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824043 with bad export cookie 14594289367152375161 [11576.239359] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11576.246853] LustreError: 408319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11576.704280] Lustre: server umount lustre-MDT0001 complete [11590.509073] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [11592.577304] Key type lgssc unregistered [11592.806880] LNet: 411312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11592.813150] LNetError: 411312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11592.833071] LNet: Removed LNI 192.168.203.127@tcp [11593.517786] Key type .llcrypt unregistered [11593.523727] Key type ._llcrypt unregistered [11613.398973] Key type ._llcrypt registered [11613.402537] Key type .llcrypt registered [11613.507157] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [11613.809291] Key type .llcrypt unregistered [11613.812930] Key type ._llcrypt unregistered [11628.052977] Key type ._llcrypt registered [11628.056525] Key type .llcrypt registered [11628.168574] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [11629.268178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11629.285185] alg: No test for adler32 (adler32-zlib) [11630.284699] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [11630.489388] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [11632.127478] Key type lgssc registered [11633.316857] Lustre: Echo OBD driver; http://www.lustre.org/ [11639.985595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11640.002701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11641.311258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11644.781399] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11651.552986] LustreError: 413545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11651.570950] LustreError: 413545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11651.746253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11651.993866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11656.973363] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11657.187385] LustreError: 413540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11657.197570] LustreError: 413540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11662.304365] LustreError: 413544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11662.330916] LustreError: 413544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11664.196917] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11666.402679] LustreError: 413540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11666.418846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11666.426167] LustreError: 413540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11671.523070] LustreError: 413545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11671.540775] LustreError: 413545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11671.638853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11672.122026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11677.263851] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11686.880113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11686.885471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11686.893174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11687.394690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11687.397909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11687.405158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11687.418792] Lustre: Skipped 1 previous similar message [11691.802068] Lustre: server umount lustre-OST0000 complete [11697.636411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11697.649959] Lustre: Skipped 1 previous similar message [11697.651197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11697.653796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11701.890032] Lustre: server umount lustre-MDT0000 complete [11702.752136] LustreError: 413545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11702.777659] LustreError: 413545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11704.969700] LustreError: 413526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824171 with bad export cookie 12214109562301278476 [11704.971262] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11704.977693] LustreError: 413526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11705.234680] Lustre: server umount lustre-MDT0001 complete [11714.353658] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [11716.801414] Key type lgssc unregistered [11717.104402] LNet: 416698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11717.143398] LNetError: 416698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11717.163980] LNet: Removed LNI 192.168.203.127@tcp [11717.821365] Key type .llcrypt unregistered [11717.823492] Key type ._llcrypt unregistered [11737.276831] Key type ._llcrypt registered [11737.278305] Key type .llcrypt registered [11737.370314] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [11738.279748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11738.359137] alg: No test for adler32 (adler32-zlib) [11739.381095] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [11739.596494] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [11741.255241] Key type lgssc registered [11742.294721] Lustre: Echo OBD driver; http://www.lustre.org/ [11748.938697] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11748.977199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11764.447571] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11769.567439] LustreError: 418068:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.127@tcp: failed processing log, type 4: rc = -110 [11796.191195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11802.234177] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11810.010205] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11817.163607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11817.688588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11817.787605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11821.566604] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11823.094489] LustreError: 418093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11823.125145] LustreError: 418093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11828.192097] LustreError: 418094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11829.327731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11829.605845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11833.720524] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11840.961826] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11847.217271] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11855.328725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11855.345468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11855.362171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11856.352663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11856.357061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11856.381070] Lustre: Skipped 1 previous similar message [11857.056822] Lustre: server umount lustre-OST0000 complete [11861.475985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11861.495126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11861.506144] Lustre: Skipped 1 previous similar message [11862.804808] Lustre: server umount lustre-MDT0000 complete [11866.153982] LustreError: 418076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824333 with bad export cookie 3574652091440416838 [11866.156579] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11866.168761] LustreError: 418076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11866.514907] Lustre: server umount lustre-MDT0001 complete [11876.766613] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [11880.234404] Key type lgssc unregistered [11880.816123] LNet: 421637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11880.836241] LNetError: 421637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11880.868341] LNet: Removed LNI 192.168.203.127@tcp [11882.104312] Key type .llcrypt unregistered [11882.109042] Key type ._llcrypt unregistered [11907.689785] Key type ._llcrypt registered [11907.691750] Key type .llcrypt registered [11907.801855] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [11908.916477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11908.937447] alg: No test for adler32 (adler32-zlib) [11909.999333] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [11910.320416] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [11912.007503] Key type lgssc registered [11913.027556] Lustre: Echo OBD driver; http://www.lustre.org/ [11919.911866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11919.936852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11921.419188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11925.452096] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11931.619064] LustreError: 423035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.631748] LustreError: 423035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11932.176726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11932.485217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11937.648859] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11946.814233] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11954.339031] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11960.383588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11960.731584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11966.717954] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11966.895565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11973.750255] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11978.721101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11978.738413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11978.757970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11981.605926] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) [11982.322822] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11982.344854] Lustre: Skipped 1 previous similar message [11984.463896] Lustre: server umount lustre-MDT0000 complete [11986.766855] LustreError: 423016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.27@tcp arrived at 1786824453 with bad export cookie 3315446630921681546 [11987.427167] LustreError: 423034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11987.456554] LustreError: 423034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11988.498651] LustreError: 423015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824455 with bad export cookie 3315446630921680650 [11988.510216] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11988.510555] LustreError: 423015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11992.544612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11992.547110] LustreError: 423031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11992.550702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11992.554830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11992.564745] LustreError: 423031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11992.593648] Lustre: Skipped 3 previous similar messages [11994.847473] Lustre: server umount lustre-OST0000 complete [11998.689539] LustreError: 423035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11998.723103] LustreError: 423035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12002.901381] Lustre: server umount lustre-MDT0001 complete [12015.823764] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [12018.192598] Key type lgssc unregistered [12018.414855] LNet: 426723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12018.422985] LNetError: 426723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12018.444601] LNet: Removed LNI 192.168.203.127@tcp [12019.157708] Key type .llcrypt unregistered [12019.160448] Key type ._llcrypt unregistered [12043.707447] Key type ._llcrypt registered [12043.709665] Key type .llcrypt registered [12043.812880] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [12044.820442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12044.921251] alg: No test for adler32 (adler32-zlib) [12045.875671] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [12046.049946] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [12047.691087] Key type lgssc registered [12048.668525] Lustre: Echo OBD driver; http://www.lustre.org/ [12055.489394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12055.504114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12056.848979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12061.611336] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12066.278699] LustreError: 428140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12066.327541] LustreError: 428140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12071.399229] LustreError: 428139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12072.103825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12072.479735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12077.010754] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12084.695579] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12090.861980] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12097.006416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12097.491734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12101.630315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12103.214654] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12109.636450] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12116.984254] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12117.015546] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12117.027325] Lustre: Skipped 1 previous similar message [12120.192673] Lustre: server umount lustre-MDT0001 complete [12122.087492] LustreError: 428868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12122.117263] LustreError: 428868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12127.201289] LustreError: 428140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12127.202703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12127.202838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12127.202842] Lustre: Skipped 1 previous similar message [12127.205722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12127.205729] Lustre: Skipped 1 previous similar message [12132.145178] Lustre: server umount lustre-OST0000 complete [12132.324097] LustreError: 428139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12136.078664] Lustre: server umount lustre-MDT0000 complete [12148.022427] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [12150.295561] Key type lgssc unregistered [12150.546705] LNet: 431828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12150.552899] LNetError: 431828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12150.563782] LNet: Removed LNI 192.168.203.127@tcp [12151.247658] Key type .llcrypt unregistered [12151.249706] Key type ._llcrypt unregistered [12175.471731] Key type ._llcrypt registered [12175.473605] Key type .llcrypt registered [12175.567519] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [12189.489557] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [12190.520276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12190.546618] alg: No test for adler32 (adler32-zlib) [12191.514773] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [12191.670965] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [12193.303555] Key type lgssc registered [12193.990842] Lustre: Echo OBD driver; http://www.lustre.org/ [12239.263900] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [12253.683299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12253.709856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12254.902128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12254.919568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12254.971638] Lustre: lustre-MDT0000: new disk, initializing [12255.066058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12255.084611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12259.711781] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12277.406782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12277.857798] Lustre: lustre-OST0000: new disk, initializing [12277.870785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12277.882961] Lustre: Skipped 1 previous similar message [12277.896152] Lustre: 436864:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12278.017655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12285.230915] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12287.698487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12287.726082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12287.806210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12297.674352] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12312.616136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12312.830892] Lustre: 435847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12312.942305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12313.005831] Lustre: lustre-MDT0001: new disk, initializing [12313.115883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12313.170328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12313.193072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12317.375288] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12325.883402] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12326.061294] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12329.753482] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12329.913856] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12335.401179] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.27@tcp (stopping) [12338.154944] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12338.165818] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12338.190220] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12338.206316] Lustre: Skipped 1 previous similar message [12340.497508] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.27@tcp (stopping) [12340.742705] Lustre: server umount lustre-MDT0001 complete [12348.392249] LustreError: 435855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12348.418256] LustreError: 435855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12353.504986] LustreError: 435853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12353.532907] LustreError: 435853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12358.625181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12358.631533] LustreError: 435858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12358.635388] Lustre: Skipped 1 previous similar message [12358.636351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12363.753317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12363.756942] LustreError: 439006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12364.267496] Lustre: server umount lustre-OST0000 complete [12374.736504] Lustre: server umount lustre-MDT0000 complete [12386.993284] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [12389.553779] Key type lgssc unregistered [12389.803599] LNet: 440614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12389.807205] LNetError: 440614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12389.814611] LNet: Removed LNI 192.168.203.127@tcp [12390.464113] Key type .llcrypt unregistered [12390.468781] Key type ._llcrypt unregistered [12429.697442] Key type ._llcrypt registered [12429.700655] Key type .llcrypt registered [12429.820839] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [12430.026734] Key type .llcrypt unregistered [12430.028465] Key type ._llcrypt unregistered [12442.784868] Key type ._llcrypt registered [12442.788223] Key type .llcrypt registered [12442.890374] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [12443.791578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12443.830223] alg: No test for adler32 (adler32-zlib) [12444.880985] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [12445.056842] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [12446.695180] Key type lgssc registered [12447.778556] Lustre: Echo OBD driver; http://www.lustre.org/ [12471.730983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12471.741841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12472.965988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12472.993616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12473.082090] Lustre: lustre-MDT0000: new disk, initializing [12473.172423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12473.185542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12477.793736] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12489.365419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12489.444448] Lustre: 443972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12489.497960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12489.502647] Lustre: Skipped 1 previous similar message [12489.551167] Lustre: lustre-MDT0001: new disk, initializing [12489.617431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12489.655245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12489.665910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12495.513907] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12505.950676] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12512.042723] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12519.441431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12519.834258] Lustre: lustre-OST0000: new disk, initializing [12519.840306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12519.846793] Lustre: 445945:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12520.036109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12526.053713] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12527.677422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12527.703813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12527.862593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12534.411527] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12540.896183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12540.902382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12540.916075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12543.459000] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12543.467772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12546.004930] Lustre: server umount lustre-MDT0000 complete [12549.594449] LustreError: 445944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825016 with bad export cookie 2993073586386950105 [12549.614472] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12550.119085] Lustre: server umount lustre-MDT0001 complete [12563.810900] Lustre: server umount lustre-OST0000 complete [12574.124909] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 16:17:19 (1786825039) [12583.349445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12583.951891] LustreError: 447971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12583.974848] LustreError: 447971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12584.050336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12588.754615] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12589.544033] LustreError: 447971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12589.578683] LustreError: 447971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12593.632851] LustreError: 447972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12598.760846] LustreError: 447971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12599.099525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12599.355180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12603.556658] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12612.027428] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12618.115508] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12638.000145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12638.208937] Lustre: Found index 0 for lustre-OST0000, updating log [12638.372588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12644.637777] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12651.128043] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12654.776575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12665.316993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12665.325748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12665.349128] LustreError: Skipped 1 previous similar message [12665.353497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12665.359263] Lustre: Skipped 2 previous similar messages [12665.387659] Lustre: Skipped 3 previous similar messages [12672.995680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12673.003718] Lustre: Skipped 2 previous similar messages [12677.087215] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12677.186521] LustreError: 449801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12677.216199] LustreError: 449801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12677.306374] Lustre: server umount lustre-OST0000 complete [12696.545426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12696.564851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12696.581653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12700.630437] Lustre: server umount lustre-MDT0000 complete [12703.713286] LustreError: 450431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12703.744084] LustreError: 450431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12704.949762] LustreError: 447952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825171 with bad export cookie 2993073586386951491 [12704.958888] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12704.967049] LustreError: 447952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12705.259684] Lustre: server umount lustre-MDT0001 complete [12709.349257] LNet: 1 local NIs in recovery (showing 1): 192.168.203.127@tcp [12716.269591] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:19:41 (1786825181) [13335.610284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13336.150420] LustreError: 454386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13336.194137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13340.251609] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13341.667438] LustreError: 454387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13346.785658] LustreError: 454386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13347.544572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13347.853637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13351.830546] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13358.852360] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13364.587315] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13383.665151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13383.977870] Lustre: Found index 0 for lustre-OST0000, updating log [13384.208512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13386.278790] LNet: 1 local NIs in recovery (showing 1): 192.168.203.127@tcp [13386.293266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13389.427913] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13395.841829] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13398.499805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13398.510960] Lustre: Skipped 1 previous similar message [13398.516920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13398.525554] Lustre: Skipped 2 previous similar messages [13403.659590] Lustre: server umount lustre-OST0000 complete [13406.704853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13406.714150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13406.727507] Lustre: Skipped 1 previous similar message [13406.748913] Lustre: Skipped 1 previous similar message [13408.864615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13408.867885] Lustre: Skipped 3 previous similar messages [13412.962601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13412.965920] Lustre: Skipped 6 previous similar messages [13413.055141] Lustre: server umount lustre-MDT0000 complete [13413.986720] LustreError: 454381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13414.011728] LustreError: 454381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13416.028118] LustreError: 455122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825882 with bad export cookie 2993073586386952583 [13416.032648] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13416.034985] LustreError: 455122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13416.392183] Lustre: server umount lustre-MDT0001 complete [13445.003968] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [13446.687155] LNet: 1 local NIs in recovery (showing 1): 192.168.203.127@tcp [13454.112413] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [13494.260776] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:32:39 (1786825959) [13513.455366] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [13515.940715] Key type lgssc unregistered [13516.185563] LNet: 461015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13516.196439] LNetError: 461015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13645.226731] LNet: Removed LNI 192.168.203.127@tcp [13646.020397] Key type .llcrypt unregistered [13646.026196] Key type ._llcrypt unregistered [13661.348480] Key type ._llcrypt registered [13661.351179] Key type .llcrypt registered [13661.502884] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [13662.682847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13662.713831] alg: No test for adler32 (adler32-zlib) [13663.780460] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [13663.986923] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [13665.663176] Key type lgssc registered [13666.569439] Lustre: Echo OBD driver; http://www.lustre.org/ [13670.045077] LNet: Added LNI 192.168.203.127@tcp100 [8/256/0/180] [13670.728821] LNet: Added LNI 192.168.203.127@tcp101 [8/256/0/180] [13671.513422] LNet: Added LNI 192.168.203.127@tcp102 [8/256/0/180] [13672.334991] LNet: Added LNI 192.168.203.127@tcp103 [8/256/0/180] [13673.073754] LNet: Added LNI 192.168.203.127@tcp104 [8/256/0/180] [13673.790897] LNet: Added LNI 192.168.203.127@tcp105 [8/256/0/180] [13674.489795] LNet: Added LNI 192.168.203.127@tcp106 [8/256/0/180] [13675.197997] LNet: Added LNI 192.168.203.127@tcp107 [8/256/0/180] [13676.053761] LNet: Added LNI 192.168.203.127@tcp108 [8/256/0/180] [13676.778621] LNet: Added LNI 192.168.203.127@tcp109 [8/256/0/180] [13677.464890] LNet: Added LNI 192.168.203.127@tcp110 [8/256/0/180] [13678.218098] LNet: Added LNI 192.168.203.127@tcp111 [8/256/0/180] [13679.053578] LNet: Added LNI 192.168.203.127@tcp112 [8/256/0/180] [13679.906906] LNet: Added LNI 192.168.203.127@tcp113 [8/256/0/180] [13680.627609] LNet: Added LNI 192.168.203.127@tcp114 [8/256/0/180] [13681.414243] LNet: Added LNI 192.168.203.127@tcp115 [8/256/0/180] [13682.144847] LNet: Added LNI 192.168.203.127@tcp116 [8/256/0/180] [13683.007963] LNet: Added LNI 192.168.203.127@tcp117 [8/256/0/180] [13683.740855] LNet: Added LNI 192.168.203.127@tcp118 [8/256/0/180] [13684.522628] LNet: Added LNI 192.168.203.127@tcp119 [8/256/0/180] [13685.319669] LNet: Added LNI 192.168.203.127@tcp120 [8/256/0/180] [13686.134798] LNet: Added LNI 192.168.203.127@tcp121 [8/256/0/180] [13686.961442] LNet: Added LNI 192.168.203.127@tcp122 [8/256/0/180] [13687.744242] LNet: Added LNI 192.168.203.127@tcp123 [8/256/0/180] [13688.392841] LNet: Added LNI 192.168.203.127@tcp124 [8/256/0/180] [13689.217168] LNet: Added LNI 192.168.203.127@tcp125 [8/256/0/180] [13689.976653] LNet: Added LNI 192.168.203.127@tcp126 [8/256/0/180] [13690.773460] LNet: Added LNI 192.168.203.127@tcp127 [8/256/0/180] [13691.604250] LNet: Added LNI 192.168.203.127@tcp128 [8/256/0/180] [13692.338055] LNet: Added LNI 192.168.203.127@tcp129 [8/256/0/180] [13693.153840] LNet: Added LNI 192.168.203.127@tcp130 [8/256/0/180] [13693.898079] LNet: Added LNI 192.168.203.127@tcp131 [8/256/0/180] [13694.708297] LNet: Added LNI 192.168.203.127@tcp132 [8/256/0/180] [13695.562194] LNet: Added LNI 192.168.203.127@tcp133 [8/256/0/180] [13696.407208] LNet: Added LNI 192.168.203.127@tcp134 [8/256/0/180] [13697.078904] LNet: Added LNI 192.168.203.127@tcp135 [8/256/0/180] [13703.543483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13703.589671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13704.688402] Lustre: 464052:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [13704.775104] LNetError: 461511:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13704.792399] LNetError: Protocol error connecting to 192.168.203.127@tcp120 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13704.810813] LNetError: Skipped 10 previous similar messages [13704.918146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13704.939531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13704.991278] LNet: 7 local NIs in recovery (showing 7): 192.168.203.127@tcp124, 192.168.203.127@tcp125, 192.168.203.127@tcp126, 192.168.203.127@tcp127, 192.168.203.127@tcp129, 192.168.203.127@tcp128, 192.168.203.127@tcp120 [13705.036463] Lustre: lustre-MDT0000: new disk, initializing [13705.116729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13705.166492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13705.825734] LNetError: 464079:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13705.839517] LNetError: 464079:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [13705.851354] LNetError: Protocol error connecting to 192.168.203.127@tcp117 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13708.072385] LNetError: 464073:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13708.084940] LNetError: 464073:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [13708.101235] LNetError: Protocol error connecting to 192.168.203.127@tcp124 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13708.109203] LNetError: Skipped 7 previous similar messages [13709.381307] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13712.160621] LNetError: 461510:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13712.171761] LNetError: Protocol error connecting to 192.168.203.127@tcp124 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13719.101630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13719.290304] Lustre: 464069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13719.336757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13719.339898] Lustre: Skipped 1 previous similar message [13719.395743] Lustre: lustre-MDT0001: new disk, initializing [13719.424690] Lustre: 464870:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [13719.430664] Lustre: 464870:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13719.518587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13719.545300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13719.559942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13720.370912] LNetError: 464082:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13720.384950] LNetError: Protocol error connecting to 192.168.203.127@tcp124 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13724.229772] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13732.377891] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13736.684935] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13736.737254] LNetError: 464079:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13736.747451] LNetError: Protocol error connecting to 192.168.203.127@tcp124 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13741.981525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13742.315393] Lustre: lustre-OST0000: new disk, initializing [13742.320028] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13742.344142] Lustre: 466044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13742.453450] Lustre: 466049:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [13742.463021] Lustre: 466049:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [13742.515241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13747.815797] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13750.332502] Lustre: 466501:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [13750.349841] Lustre: 466501:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13750.408955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13750.420310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13750.541348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13753.056793] LNetError: 461510:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13753.083189] LNetError: 461510:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [13753.118308] LNetError: Protocol error connecting to 192.168.203.127@tcp122 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13753.128995] LNetError: Skipped 8 previous similar messages [13756.586564] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13764.384216] LNet: 1 local NIs in recovery (showing 1): 192.168.203.127@tcp124 [13766.112586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13766.120993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13766.121705] LustreError: Skipped 1 previous similar message [13766.127962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13766.165037] Lustre: Skipped 1 previous similar message [13766.180925] Lustre: Skipped 1 previous similar message [13768.808895] Lustre: server umount lustre-OST0000 complete [13775.840305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13775.844351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13775.864569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13778.600877] Lustre: server umount lustre-MDT0000 complete [13782.064173] LustreError: 464062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826248 with bad export cookie 7685336369916452132 [13782.068043] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13782.073425] LustreError: 464062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13782.433122] Lustre: server umount lustre-MDT0001 complete [13791.919737] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [13792.995360] LNetError: 461509:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13793.004334] LNetError: 461509:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 15 previous similar messages [13793.010935] LNetError: Protocol error connecting to 192.168.203.127@tcp117 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13793.025970] LNetError: Skipped 15 previous similar messages [13794.783808] Key type lgssc unregistered [13795.151843] LNet: 467961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13795.157908] LNetError: 467961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13795.165946] LNet: Removed LNI 192.168.203.127@tcp [13796.191486] LNet: Removed LNI 192.168.203.127@tcp100 [13796.199445] LNetError: 467961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13796.213029] LNetError: 467961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13796.227627] LNet: Removed LNI 192.168.203.127@tcp101 [13797.279142] LNet: Removed LNI 192.168.203.127@tcp102 [13797.285299] LNetError: 467961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13797.292676] LNetError: 467961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13797.301746] LNet: Removed LNI 192.168.203.127@tcp103 [13798.367153] LNet: Removed LNI 192.168.203.127@tcp104 [13799.391169] LNet: Removed LNI 192.168.203.127@tcp105 [13799.398352] LNetError: 467961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13799.406404] LNetError: 467961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13799.413461] LNet: Removed LNI 192.168.203.127@tcp106 [13800.479155] LNet: Removed LNI 192.168.203.127@tcp107 [13801.503285] LNet: Removed LNI 192.168.203.127@tcp108 [13802.530363] LNet: Removed LNI 192.168.203.127@tcp109 [13803.551245] LNet: Removed LNI 192.168.203.127@tcp110 [13803.554110] LNetError: 467961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13803.568137] LNetError: 467961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13803.584683] LNet: Removed LNI 192.168.203.127@tcp111 [13804.639152] LNet: Removed LNI 192.168.203.127@tcp112 [13805.663180] LNet: Removed LNI 192.168.203.127@tcp113 [13806.687254] LNet: Removed LNI 192.168.203.127@tcp114 [13807.711742] LNet: Removed LNI 192.168.203.127@tcp115 [13808.736231] LNet: Removed LNI 192.168.203.127@tcp116 [13809.759694] LNet: Removed LNI 192.168.203.127@tcp117 [13810.783256] LNet: Removed LNI 192.168.203.127@tcp118 [13811.807550] LNet: Removed LNI 192.168.203.127@tcp119 [13811.818724] LNetError: 467961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13811.827875] LNetError: 467961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13811.835922] LNet: Removed LNI 192.168.203.127@tcp120 [13812.895131] LNet: Removed LNI 192.168.203.127@tcp121 [13813.919554] LNet: Removed LNI 192.168.203.127@tcp122 [13814.943530] LNet: Removed LNI 192.168.203.127@tcp123 [13814.946898] LNet: Removed LNI 192.168.203.127@tcp124 [13815.967187] LNet: Removed LNI 192.168.203.127@tcp125 [13816.991257] LNet: Removed LNI 192.168.203.127@tcp126 [13818.015709] LNet: Removed LNI 192.168.203.127@tcp127 [13819.043864] LNet: Removed LNI 192.168.203.127@tcp128 [13820.063160] LNet: Removed LNI 192.168.203.127@tcp129 [13820.065891] LNet: Removed LNI 192.168.203.127@tcp130 [13820.071853] LNet: Removed LNI 192.168.203.127@tcp131 [13820.078644] LNet: Removed LNI 192.168.203.127@tcp132 [13820.083491] LNet: Removed LNI 192.168.203.127@tcp133 [13820.088478] LNet: Removed LNI 192.168.203.127@tcp134 [13820.097972] LNet: Removed LNI 192.168.203.127@tcp135 [13821.139099] Key type .llcrypt unregistered [13821.141352] Key type ._llcrypt unregistered [13847.062802] Key type ._llcrypt registered [13847.065691] Key type .llcrypt registered [13847.208732] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [13859.813029] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [13860.940261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13861.074160] alg: No test for adler32 (adler32-zlib) [13862.105173] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [13862.299471] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [13863.975164] Key type lgssc registered [13865.137213] Lustre: Echo OBD driver; http://www.lustre.org/ [13900.475402] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:39:25 (1786826365) [13916.604150] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [13918.936917] Key type lgssc unregistered [13919.137857] LNet: 472033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13919.146875] LNetError: 472033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13919.162470] LNet: Removed LNI 192.168.203.127@tcp [13919.882357] Key type .llcrypt unregistered [13919.889820] Key type ._llcrypt unregistered [13935.939509] Key type ._llcrypt registered [13935.941781] Key type .llcrypt registered [13936.020340] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [13937.119025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13937.127767] alg: No test for adler32 (adler32-zlib) [13938.180982] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [13938.438075] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [13940.104929] Key type lgssc registered [13941.160907] Lustre: Echo OBD driver; http://www.lustre.org/ [13949.471326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13949.518762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13950.671934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13950.711520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13950.779586] Lustre: lustre-MDT0000: new disk, initializing [13950.853500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13950.874883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13954.542905] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13963.528873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13963.667149] Lustre: 473254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13963.731919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13963.736711] Lustre: Skipped 1 previous similar message [13963.806266] Lustre: lustre-MDT0001: new disk, initializing [13963.940561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13963.989204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13964.004780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13968.113713] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13976.813580] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13982.048752] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13987.780467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13988.065965] Lustre: lustre-OST0000: new disk, initializing [13988.072884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13988.078736] Lustre: 475222:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13988.190044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13993.051435] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13995.606680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13995.623839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13995.766808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14001.740946] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14004.884123] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing lnet_if_list [14010.259938] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14010.295867] LNet: Added LNI 192.168.203.127@tcp10 [8/256/0/180] [14015.590725] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14015.632134] LNet: Added LNI 192.168.203.127@tcp11 [8/256/0/180] [14021.335533] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14021.407682] LNet: Added LNI 192.168.203.127@tcp12 [8/256/0/180] [14026.199588] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14026.236032] LNet: Added LNI 192.168.203.127@tcp13 [8/256/0/180] [14031.949675] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14032.011526] LNet: Added LNI 192.168.203.127@tcp14 [8/256/0/180] [14037.597683] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14037.620149] LNet: Added LNI 192.168.203.127@tcp15 [8/256/0/180] [14044.108845] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14044.161425] LNet: Added LNI 192.168.203.127@tcp16 [8/256/0/180] [14050.373841] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14050.412634] LNet: Added LNI 192.168.203.127@tcp17 [8/256/0/180] [14055.925076] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14055.950512] LNet: Added LNI 192.168.203.127@tcp18 [8/256/0/180] [14059.677628] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14059.711163] LNet: Added LNI 192.168.203.127@tcp19 [8/256/0/180] [14064.549358] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14064.580833] LNet: Added LNI 192.168.203.127@tcp20 [8/256/0/180] [14068.699282] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14068.734240] LNet: Added LNI 192.168.203.127@tcp21 [8/256/0/180] [14073.836336] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14073.862667] LNet: Added LNI 192.168.203.127@tcp22 [8/256/0/180] [14078.734512] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14078.755076] LNet: Added LNI 192.168.203.127@tcp23 [8/256/0/180] [14083.993986] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14084.029385] LNet: Added LNI 192.168.203.127@tcp24 [8/256/0/180] [14089.919228] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14089.964877] LNet: Added LNI 192.168.203.127@tcp25 [8/256/0/180] [14095.552737] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14095.575654] LNet: Added LNI 192.168.203.127@tcp26 [8/256/0/180] [14101.883732] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14101.906253] LNet: Added LNI 192.168.203.127@tcp27 [8/256/0/180] [14107.604786] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14107.635647] LNet: Added LNI 192.168.203.127@tcp28 [8/256/0/180] [14112.680361] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14112.708575] LNet: Added LNI 192.168.203.127@tcp29 [8/256/0/180] [14118.823426] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14118.860165] LNet: Added LNI 192.168.203.127@tcp30 [8/256/0/180] [14124.612628] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14124.634512] LNet: Added LNI 192.168.203.127@tcp31 [8/256/0/180] [14131.041343] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14131.070380] LNet: Added LNI 192.168.203.127@tcp32 [8/256/0/180] [14137.367722] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14137.393305] LNet: Added LNI 192.168.203.127@tcp33 [8/256/0/180] [14143.976044] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14144.024862] LNet: Added LNI 192.168.203.127@tcp34 [8/256/0/180] [14150.169574] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14150.203311] LNet: Added LNI 192.168.203.127@tcp35 [8/256/0/180] [14156.476942] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14156.518389] LNet: Added LNI 192.168.203.127@tcp36 [8/256/0/180] [14162.871773] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14162.895830] LNet: Added LNI 192.168.203.127@tcp37 [8/256/0/180] [14169.131217] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14169.157226] LNet: Added LNI 192.168.203.127@tcp38 [8/256/0/180] [14175.212860] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14175.242713] LNet: Added LNI 192.168.203.127@tcp39 [8/256/0/180] [14181.320247] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14181.341254] LNet: Added LNI 192.168.203.127@tcp40 [8/256/0/180] [14187.790427] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14187.813459] LNet: Added LNI 192.168.203.127@tcp41 [8/256/0/180] [14193.846101] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14193.876372] LNet: Added LNI 192.168.203.127@tcp42 [8/256/0/180] [14200.087754] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14200.110857] LNet: Added LNI 192.168.203.127@tcp43 [8/256/0/180] [14205.845696] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14205.870721] LNet: Added LNI 192.168.203.127@tcp44 [8/256/0/180] [14212.268932] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14212.295437] LNet: Added LNI 192.168.203.127@tcp45 [8/256/0/180] [14219.532678] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14219.559500] LNet: Added LNI 192.168.203.127@tcp46 [8/256/0/180] [14227.517843] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14227.546364] LNet: Added LNI 192.168.203.127@tcp47 [8/256/0/180] [14236.082503] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14236.111708] LNet: Added LNI 192.168.203.127@tcp48 [8/256/0/180] [14245.113347] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14245.141659] LNet: Added LNI 192.168.203.127@tcp49 [8/256/0/180] [14253.559699] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14253.595478] LNet: Added LNI 192.168.203.127@tcp50 [8/256/0/180] [14270.438408] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14270.443204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14270.448902] Lustre: Skipped 2 previous similar messages [14270.462781] Lustre: Skipped 2 previous similar messages [14275.554219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14275.561137] Lustre: Skipped 2 previous similar messages [14275.811771] Lustre: server umount lustre-MDT0000 complete [14279.374236] LustreError: 473245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826746 with bad export cookie 15516415397938201261 [14279.380267] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14279.394212] LustreError: 473245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14279.771884] Lustre: server umount lustre-MDT0001 complete [14296.864107] Lustre: 472526:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786826747/real 1786826747] req@000000007b717ed1 x1873623284614656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786826763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14296.909496] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14297.118669] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [14300.129815] Lustre: 472524:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786826751/real 1786826751] req@00000000a9eab929 x1873623284614784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786826767 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14301.215104] Lustre: 472524:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786826752/real 1786826752] req@000000005644e157 x1873623284614912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786826768 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14304.950796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14305.247245] Lustre: 472527:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786826756/real 1786826756] req@000000006c034a1c x1873623284615168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786826772 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14310.431279] Lustre: 472527:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786826761/real 1786826761] req@00000000f308a252 x1873623284615296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786826777 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14324.856280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14325.869508] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14330.540522] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14332.396214] LustreError: 475214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14332.443432] LustreError: 475214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14337.525979] LustreError: 475215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14340.099272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14340.434529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14341.486388] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14344.715385] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14352.234953] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14357.538428] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14376.417370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14376.417717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14376.427824] LustreError: Skipped 1 previous similar message [14376.428788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14376.453956] Lustre: Skipped 1 previous similar message [14381.508698] Lustre: server umount lustre-OST0000 complete [14386.655820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14386.665701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14386.680270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14386.688053] Lustre: Skipped 3 previous similar messages [14391.311253] Lustre: server umount lustre-MDT0000 complete [14392.804799] LustreError: 499462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14392.817042] LustreError: 499462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14394.553572] LustreError: 475221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826861 with bad export cookie 15516415397938203858 [14394.561111] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14394.563756] LustreError: 475221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14394.836927] Lustre: server umount lustre-MDT0001 complete [14404.808332] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [14407.495385] Key type lgssc unregistered [14407.836646] LNet: 502242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14407.843899] LNetError: 502242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14407.852895] LNet: Removed LNI 192.168.203.127@tcp [14407.856436] LNet: Removed LNI 192.168.203.127@tcp10 [14407.860312] LNet: Removed LNI 192.168.203.127@tcp11 [14407.868464] LNet: Removed LNI 192.168.203.127@tcp12 [14407.872750] LNet: Removed LNI 192.168.203.127@tcp13 [14407.877625] LNet: Removed LNI 192.168.203.127@tcp14 [14407.883337] LNet: Removed LNI 192.168.203.127@tcp15 [14407.886448] LNet: Removed LNI 192.168.203.127@tcp16 [14407.890866] LNet: Removed LNI 192.168.203.127@tcp17 [14407.894333] LNet: Removed LNI 192.168.203.127@tcp18 [14407.898450] LNet: Removed LNI 192.168.203.127@tcp19 [14407.901353] LNet: Removed LNI 192.168.203.127@tcp20 [14407.903980] LNet: Removed LNI 192.168.203.127@tcp21 [14407.907485] LNet: Removed LNI 192.168.203.127@tcp22 [14407.921498] LNet: Removed LNI 192.168.203.127@tcp23 [14407.926400] LNet: Removed LNI 192.168.203.127@tcp24 [14407.929721] LNet: Removed LNI 192.168.203.127@tcp25 [14407.932965] LNet: Removed LNI 192.168.203.127@tcp26 [14407.935982] LNet: Removed LNI 192.168.203.127@tcp27 [14407.939134] LNet: Removed LNI 192.168.203.127@tcp28 [14407.942722] LNet: Removed LNI 192.168.203.127@tcp29 [14407.945886] LNet: Removed LNI 192.168.203.127@tcp30 [14407.951638] LNet: Removed LNI 192.168.203.127@tcp31 [14407.954908] LNet: Removed LNI 192.168.203.127@tcp32 [14407.958847] LNet: Removed LNI 192.168.203.127@tcp33 [14407.962458] LNet: Removed LNI 192.168.203.127@tcp34 [14407.965482] LNet: Removed LNI 192.168.203.127@tcp35 [14407.968795] LNet: Removed LNI 192.168.203.127@tcp36 [14407.973423] LNet: Removed LNI 192.168.203.127@tcp37 [14407.983712] LNet: Removed LNI 192.168.203.127@tcp38 [14407.992131] LNet: Removed LNI 192.168.203.127@tcp39 [14407.997287] LNet: Removed LNI 192.168.203.127@tcp40 [14408.004874] LNet: Removed LNI 192.168.203.127@tcp41 [14408.009333] LNet: Removed LNI 192.168.203.127@tcp42 [14408.012285] LNet: Removed LNI 192.168.203.127@tcp43 [14408.016488] LNet: Removed LNI 192.168.203.127@tcp44 [14408.020053] LNet: Removed LNI 192.168.203.127@tcp45 [14408.023203] LNet: Removed LNI 192.168.203.127@tcp46 [14408.028308] LNet: Removed LNI 192.168.203.127@tcp47 [14408.031720] LNet: Removed LNI 192.168.203.127@tcp48 [14408.034977] LNet: Removed LNI 192.168.203.127@tcp49 [14408.044973] LNet: Removed LNI 192.168.203.127@tcp50 [14408.676562] Key type .llcrypt unregistered [14408.679598] Key type ._llcrypt unregistered [14426.018993] Key type ._llcrypt registered [14426.021137] Key type .llcrypt registered [14426.093244] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [14437.443147] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [14438.310101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14438.370260] alg: No test for adler32 (adler32-zlib) [14439.346581] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [14439.582113] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [14441.247503] Key type lgssc registered [14442.103568] Lustre: Echo OBD driver; http://www.lustre.org/ [14469.458753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14469.479356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14470.799544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14470.870747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14470.972120] Lustre: lustre-MDT0000: new disk, initializing [14471.076413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14471.091167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14474.824140] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14484.403509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14484.480821] Lustre: 505417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14484.536336] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14484.540796] Lustre: Skipped 1 previous similar message [14484.612732] Lustre: lustre-MDT0001: new disk, initializing [14484.696467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14484.742932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14484.751575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14488.601616] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14496.502664] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14500.815451] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14506.461765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14506.581620] Lustre: *** cfs_fail_loc=90f, val=0*** [14506.633166] Lustre: lustre-OST0000: new disk, initializing [14506.636713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14506.642859] Lustre: 507431:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14506.704185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14512.175397] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14515.771205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14515.782473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14515.885410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14519.780339] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14526.437096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14526.438306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14526.443152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14526.469416] Lustre: Skipped 1 previous similar message [14531.425698] Lustre: server umount lustre-OST0000 complete [14536.162833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14536.175231] LustreError: Skipped 1 previous similar message [14536.179802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14536.190967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14536.195059] Lustre: Skipped 1 previous similar message [14540.742997] Lustre: server umount lustre-MDT0000 complete [14543.752432] LustreError: 505408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827010 with bad export cookie 2724611928839427619 [14543.759289] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14543.764203] LustreError: 505408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14544.103242] Lustre: server umount lustre-MDT0001 complete [14551.746294] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [14553.960647] Key type lgssc unregistered [14554.221414] LNet: 509337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14554.228735] LNetError: 509337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14554.243129] LNet: Removed LNI 192.168.203.127@tcp [14554.959391] Key type .llcrypt unregistered [14554.962388] Key type ._llcrypt unregistered [14569.877229] Key type ._llcrypt registered [14569.878933] Key type .llcrypt registered [14570.013964] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [14570.719738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14570.800576] alg: No test for adler32 (adler32-zlib) [14571.816562] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [14571.991515] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [14573.623289] Key type lgssc registered [14574.483205] Lustre: Echo OBD driver; http://www.lustre.org/ [14580.023348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14580.035805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14581.337916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14584.634082] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14590.657808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14590.933128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14594.325995] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14599.395217] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14603.564698] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14608.947213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14609.409948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14615.419642] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14621.819572] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14634.516827] loop0: detected capacity change from 0 to 204800000 [14634.685787] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14634.700471] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14634.719372] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14639.813573] loop0: detected capacity change from 0 to 204800000 [14640.005619] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14640.023071] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14640.042546] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14643.895882] loop0: detected capacity change from 0 to 204800000 [14648.439175] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14648.588817] Lustre: fs15246-OST0000: new disk, initializing [14648.593779] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14648.600912] Lustre: 514175:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14648.659817] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14653.834754] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14662.263788] loop1: detected capacity change from 0 to 204800000 [14667.504055] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14667.609244] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14667.681529] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14667.790668] Lustre: fs15246-MDT0000: new disk, initializing [14668.045476] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14668.079729] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14668.274479] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14669.351586] Lustre: 515298:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000056f8fef5 x1873623948243328/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786827152 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14672.383875] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14678.442478] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14678.458974] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14678.496443] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14692.856452] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14692.874410] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14697.955852] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14698.925536] Lustre: server umount fs15246-MDT0000 complete [14703.081814] LustreError: 510582:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14703.100557] LustreError: 510582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14707.017289] Lustre: server umount fs15246-OST0000 complete [14717.920857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14717.932737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14717.951416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14723.552528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14723.560545] Lustre: Skipped 3 previous similar messages [14728.673236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14728.680348] Lustre: Skipped 1 previous similar message [14731.240170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14731.570827] Lustre: server umount lustre-OST0000 complete [14738.912889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14738.915216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14738.942171] Lustre: Skipped 2 previous similar messages [14738.957197] Lustre: Skipped 1 previous similar message [14742.686367] Lustre: server umount lustre-MDT0000 complete [14744.032580] LustreError: 511315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14744.047626] LustreError: 511315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14746.531715] LustreError: 517439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827213 with bad export cookie 14975505664611525084 [14746.533139] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14746.555072] LustreError: 517439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14747.126731] Lustre: server umount lustre-MDT0001 complete [14757.386035] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [14759.494375] Key type lgssc unregistered [14759.715489] LNet: 518396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14759.721863] LNetError: 518396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14759.735553] LNet: Removed LNI 192.168.203.127@tcp [14760.342716] Key type .llcrypt unregistered [14760.344830] Key type ._llcrypt unregistered [14774.801172] Key type ._llcrypt registered [14774.802276] Key type .llcrypt registered [14774.918715] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [14775.524093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14775.566028] alg: No test for adler32 (adler32-zlib) [14776.701459] Lustre: Lustre: Build Version: 2.17.57_2_gb7122c1 [14777.019825] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [14778.896276] Key type lgssc registered [14780.439276] Lustre: Echo OBD driver; http://www.lustre.org/ [14815.498280] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [14823.116166] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [14853.909384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14853.925783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14855.186290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14855.241573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14855.372147] Lustre: lustre-MDT0000: new disk, initializing [14855.470588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14855.502437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14859.348085] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14867.772844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14867.966193] Lustre: 522841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14868.054866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14868.059698] Lustre: Skipped 1 previous similar message [14868.158681] Lustre: lustre-MDT0001: new disk, initializing [14868.249548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14868.298214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14868.312842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14873.197808] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14882.662603] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14887.989373] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14894.357521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14894.626678] Lustre: lustre-OST0000: new disk, initializing [14894.630190] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14894.636820] Lustre: 524808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14894.714401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14900.310402] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14902.993336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14903.023764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14903.311707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14912.466473] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14919.066265] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14919.335969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14923.993753] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14924.256085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14928.874296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14928.881457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14928.894222] LustreError: Skipped 1 previous similar message [14928.895511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14928.922379] Lustre: Skipped 1 previous similar message [14928.961385] Lustre: Skipped 1 previous similar message [14932.496109] Lustre: server umount lustre-OST0000 complete [14939.106240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14939.112584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14939.131416] Lustre: Skipped 1 previous similar message [14939.151413] Lustre: Skipped 1 previous similar message [14944.224850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14944.228527] Lustre: Skipped 1 previous similar message [14948.321278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14948.329922] Lustre: Skipped 1 previous similar message [14951.395329] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14951.605917] Lustre: server umount lustre-MDT0000 complete [14955.545952] LustreError: 522832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827422 with bad export cookie 15450799955720315992 [14955.547735] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14955.561158] LustreError: 522832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14955.880517] Lustre: server umount lustre-MDT0001 complete [14979.811380] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [14988.170973] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [15022.847953] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14769 sec ========= 16:58:08 (1786827488) [15024.260746] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:58:09 (1786827489) === [15027.818111] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:58:13 (1786827493) === [15065.470072] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [15067.888916] Key type lgssc unregistered [15068.189191] LNet: 531461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15068.200408] LNetError: 531461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15068.211917] LNet: Removed LNI 192.168.203.127@tcp [15068.988796] Key type .llcrypt unregistered [15068.991078] Key type ._llcrypt unregistered